Trend and Theme Detection in Financial Market Knowledge
A common request we get at Noonum is to show prospects what we can infer about emerging trends and themes. Most potential customers who ask this question often have a team of research analysts trying to find ways to stay on top of an endless stream and ever-growing mountain of information that could impact their coverage universe. Can we help them sift through this data, identify what’s actionable, and drive better investment decisions?
Reading is how we acquire knowledge but how do we identify patterns and insights.
This use case for Noonum is about primarily about improving the process and effectiveness of market research, or knowledge acquisition. Distilling insights from unstructured text is a grueling process. Anyone involved in market research appreciates that collecting and ingesting the right content is just the first step. From there, the relevant securities, companies, products, people, and places need to be identified, extracted, and structured. The expert market researcher uses diverse sources of data to identify patterns, and thus an AI system to help that expert must also consider them.
By combining our graph of relationships extracted from millions of texts with this other structured data to indicate stock performance, corporate activities and global events, it’s possible to try and build correlation and causal graph representations and run “what if” scenarios for future events. For experts in the field, this skill comes from experience: over the years having read a lot and acquired a lot of knowledge, and from witnessing market and corporate activities alongside various geopolitical and macroeconomic events, experts can align new knowledge into patterns and see analogies and likely cause and effect chains that others cannot connect the dots across.
Translate reading to an AI task for market analysis.
How do we translate the above market research task into something that AI can help with and assist experts to see even more accurate or complex patterns? Well, we can map the above process an expert uses into our AI system: structure knowledge from various sources into a graphical representation of things, their relationships to one another and to when they occurred. We then add in the strength of predictions made when reading the text with NLP models to create a graph where “hypothesis” generation and validation can be performed. Either via the user or with additional inference algorithms, the graph can be used to build specific causal graphs around events like mergers and acquisitions, IPOs, significant market movements. As a library of hypotheses is developed over time, they can be checked with new data for accuracy and relevancy, and the most resilient or confident are identified.
AI needs specialization as opposed to generalization.
The reality, and challenge, of the above “insight” machine is that every expert, in every industry and at every different point in time uses a slightly different relative and performant internal insight machine. A junior analyst straight out of school brings an appreciation of new technologies like social media as well as a lack of direct experience to previous market events, so their insight machine is going to be very different than a senior analyst. And that’s a good thing - organizations and teams depend on this diversity. But many AI systems and models tend to not appreciate diversity and specialization, instead, they focus on identifying all patterns from the largest available data sets. These AI systems may be good at detecting some broad pattern but they do not resemble what the expert in market research is good at: it’s why most experts cover niche areas of technologies or industries; they need to specialization their approach and knowledge specific data and types of insights.
When we set out to help customers identify trends and topics, we have to consider how experts do this process today. Firstly, we work to bring market segmentation into account in a similar way that research analysts cover niche markets. Different industries operate at different time scales and over different amounts of knowledge. Our hypotheses need to consider that industries add a specialized set of thresholds that must be dynamic across different industries and over time periods. Trends and topics are relevant within industry groups and also within the size of organizations, determined with metrics like the market capitalization.
Secondly, an analyst doesn’t read every information source, they focus on unique, trusted and important sources. But our AI system can read nearly every source. So, we must deploy a series of deduplication to avoid biases from syndicated or repeated news across sources as well as carefully curate and weight different sources over time.
Lastly, topics, or themes, themselves are very unique: some themes like “climate change” get very different coverage than “warehouse robotics” or “educational technology”, so how themes are identified as trending must also be very theme specific and dynamic over time. To accomplish this, we use a variety of statistical methods that are sound in identifying significance for correlation, anomaly detection, and trend identification. These methods provide a sound way to identify trend and themes that can be dynamic and responsive to specific themes and industries at different periods in time - without setting specific thresholds or weights that become inaccurate later or do not apply to different themes or industries.
Through assistance and architecting, AI complements the expert for insight generation.
In conclusion, using a graph of structured data and their relationships captured in a time series, where prediction strengths and additional inferences are layered upon, we can help the expert semi-automatically identify trends and themes that are emerging across a diverse set of data, markets and events.
