Category: Uncategorized

  • The road not taken – exploring ‘what if?’ with causal analytics

    As organisations increasingly focus on measuring the outcomes of their decisions, recent developments in causal analytics are helping decision-makers evaluate the decisions they have made in the past to improve the decisions they will make in the future.

    In 2022, the electronics manufacturer Hisense sponsored the FIFA World Cup and invested a significant amount in getting the Hisense brand associated with the massive global event. Naturally, Hisense would want to quantify the return on investment in the form of brand awareness and sales.

    Measuring impact can be complex

    With the increasing availability of large amounts of data, organisations like Hisense have become more aware of the competitive advantages of measuring the impact of marketing campaigns – but the world of marketing is diverse and complex. The impacts of routine direct marketing activities through owned channels, such as direct-marketing email campaigns, can be easily tracked with well-established statistical methods. The effect of broader based campaigns, such as mass media campaigns across an entire country or the world (like sponsoring the FIFA World Cup), are much more difficult to attribute back to the marketing.

    Seeing the unseen

    The crux of this challenge lies with identifying a ‘what if?’ world, where you can see what would have happened if a marketing intervention was not implemented. In the case of direct marketing, this can be done with carefully constructed control groups.

    But what do we do when it’s not possible to construct matched controls, as in Hisense’s case? Establishing the business benefit from its investment decision is inherently difficult, as it requires comparison to a scenario which did not occur – one where Hisense chose not to sponsor the World Cup.

    The question of ‘what if?’ lies at the heart of evaluating the impact of most high-cost, high-risk decisions. What if we had decided not to proceed with this marketing campaign? What if we had decided not to enact this government policy?

    Causal analytics help to capture the impacts of major campaigns, like sponsoring the FIFA World Cup

    Lifting the veil on ‘what if?’

    Recently, advances in the field of causal analysis are lifting the veil on ‘what if?’. We can now take a peek at alternate counterfactual scenarios, even where direct controls are not possible, helping us to put firmer estimates on the returns from broad-based marketing activities. Advanced analytics is allowing us to explore the road not taken.

    For Hisense, this means we can reasonably answer questions such as Can we see evidence of the effect of its near-blanket coverage across the largest sporting event in the world on brand sentiment? How large was the effect? Did the effect persist beyond the World Cup and, if so, for how long?

    Understanding what would have happened

    If determining the impact, or causal effect, of a decision requires us to understand what would have happened in the absence of that decision, we can do this by calculating a ‘treatment effect’. This is defined as a comparison between the outcomes of a group impacted by an intervention (for example a marketing campaign) compared to the outcomes in a control group with no intervention.

    In some scenarios, a ‘randomised controlled trial’ (RCT), can help determine a treatment effect. In an RCT, a random selection of participants receives the intervention (the treatment group) while other participants do not (the control group). Because the allocation of the intervention is random, it can be reasonably asserted that the difference in the outcomes between the treatment and control groups is mostly due to the intervention, and not due to other ‘confounding’ factors.

    However, there are many scenarios where an RCT is not applicable. Sometimes, defining a treatment and control group could be unethical, for example in instituting government policy. In other situations, it might simply be impossible. Hisense cannot designate a control group for its World Cup sponsorship as it is a global event watched by a significant proportion of the world population.

    How ‘mental availability’ offers the first clues

    A plausible first step to finding answers is to identify and measure proxies for ‘mental availability’. This refers to a brand coming to mind as a potential customer considers a purchase.

    A credible proxy for this measure is the level of search interest on Google, where we assume consumers actively searching for a brand reflects mental availability. Google search statistics are available for any topic and any region through the Google Trends service. In the graph below, we show the worldwide Trends data for the search term ‘Hisense’ from January 2021 to March of 2023.

    How can we tell if the surge in interest is due to the World Cup? A synthetic control holds the key

    It is apparent that Hisense had a large increase in Google search activity in the period around the World Cup in November of 2022, but this coincides with another large event in the marketing calendar – Black Friday on 25 November. Moreover, we can see several surges in interest in previous time periods. What caused these surges? Is the post-World Cup increase a direct, causal outcome of the World Cup? Or is it just part of the normal rhythms of the universal sales calendar, such as Black Friday and Christmas sales? To answer these questions, and in the absence of a ‘matched’ control, we need a robust ‘representative’ baseline control – enter the synthetic control.

    The synthetic control – something out of nothing

    If it is not feasible to have a real life, ‘organic’ control, then one possible solution is to have an artificially constructed post-hoc ‘synthetic’ control. Synthetic controls approximate a real control by piecing together information from other similar groups not impacted by the intervention.

    For example, it is plausible that the brand awareness of Hisense at any given time could be approximated by taking a weighted sum of the brand awareness of other similar competitor brands. And, further, given no competitors sponsored the World Cup, it could then be possible to simulate an alternate reality where Hisense also did not sponsor the World Cup.

    Kicking goals with Structural Time Series

    An analytical technique that has received recent prominence for tackling this problem is the Bayesian Structural Time Series model. These models are highly customisable and can ingest a variety of different factors to approximate a time series, including trends, seasonal effects and dependent variables.

    The significant advantage that Bayesian Structural models have over other time series modelling techniques is that they are explainable, predominantly for two reasons. Firstly, because they are made up of constituent ‘building blocks’, it is easy to understand the underlying dynamics of how the observation is changing over time. Secondly, they inherently generate an estimation of uncertainty, which gives a measure of how confident you should be in the prediction.

    We can extract search trends for several electronics retailers, including Samsung, Sony and LG, to be used to create the synthetic control. The general movements of the curves are in good alignment in the pre-World Cup period – when the interest levels of a brand rise typically the interest levels of the other brands also rise. This suggests that the competitor brands are good candidates to be used as building blocks in the Bayesian Structural models.

    Extracting trends for similar brands helps us better attribute the surge in interest for Hisense

    Applying the Bayesian Structural Time Series technique1,2 to the Google search data for the competitor brands yields a synthetic control for Hisense interest levels. In addition to an overall predicted synthetic control, the algorithm also produces representations of the building blocks or components, which we illustrate below.

    This shows rising interest in Hisense in addition to effects not captured by the comparison brands

    Effects from interest levels of other brands. Together (a) and (b) create our synthetic control

    The first component (a) represents the trends and natural volatility in the Hisense interest levels. This captures the general increasing trend in the Google search interest of Hisense over time in addition to any volatile effects that are not captured by the interest levels in the other brands. The second component (b) represents the effects that can be extracted from the interest levels of other brands, including seasonal sales, general consumer sentiment, consumer expos and more. These two curves are blended to create the overall predicted synthetic control curve.

    Big sponsorship impact, shown by actual interest levels above synthetic levels after the World Cup

    The trend estimates from the model approximates the real Hisense Google interest reasonably accurately before the World Cup, even in times of disruption like Black Friday of 2021. This gives some comfort that the synthetic control will appropriately simulate the alternate reality where Hisense did not sponsor the World Cup.

    Comparing the synthetic control and actual search data yields interesting results. Clearly the actual interest levels are well above the synthetic interest levels following the World Cup – indicating a strong impact from the sponsorship. Overall, across the World Cup period, the middle estimate of the change in search interest of the Hisense brand on Google is a healthy increase of 22%.

    Overall, the change in interest for Hisense across the World Cup period is a healthy 22% increase

    The cumulative benefit increases rapidly at first, coinciding with the initial excitement of the opening games of the tournament, and then begins to tail off as the festivities draw to a close, eventually falling back to the baseline in January 2023. The return to the baseline poses an important question: is the increased brand awareness in Hisense transient or is it long lasting? At first glance, it appears to be transient, but a longer observational period into the future is required to assess properly. Clearly, assessing the long-term impact of the sponsorship is vital to assessing value for money.

    A force for good

    Causal analysis is a rapidly evolving area, with new developments in the field being discussed in earnest within the analytics community. Although much of the new interest has been driven by a greater desire within the corporate world to understand the impact of business decisions, understanding causal relationships is also vitally important in evaluating public policy.

    Synthetic controls have proven especially useful in the government space – so much so that the technique has been described as “arguably the most important innovation in public policy literature in the last 15 years” by 2021 Nobel laureate Professor Guido W. Imbens and causal analysis expert Professor Susan Athey3.

    Ultimately, exploring the road not taken – those ‘what if?’ scenarios – is an enormous step towards making better decisions –in terms of bolstering the benefits in business and improving outcomes for members of the community. And this is just the beginning. There is much potential to achieve this complementary effect across many industries to make a difference now and well into the future.

    Citations
    1. Brodersen, K.H., Gallusser, F., Koehler, J., Remy, N. and Scott, S.L., 2015. Inferring causal impact using Bayesian structural time-series models.The Annals of Applied Statistics, pp.247-274.
    2. Causal Impact 1.3.0, Brodersen et al., Annals of Applied Statistics (2015). https://google.github.io/CausalImpact/
    3. Athey, Susan, and Guide W. Imbens. 2017a. “The State of Applied Econometrics: Causality and Policy Evaluation.” Journal of Economic Perspectives 31 (2): 3–32.
  • Building cyber resilience – 4 critical steps for boards

    The most urgent issues in cybersecurity affecting organisations today are complex and often perplexing for organisations. An Actuaries Institute panel of experts and thought leaders came together recently to help navigate a pragmatic and effective path forward.

    The cyberattacks on Optus, Medibank and Latitude Financial are growing evidence at home of the increasing sophistication of hackers across the globe. As criminals scan cyber space for any vulnerability, it’s clear no organisation is immune and the need for protection is more important than ever. But how can organisations and particularly boards build their cyber mettle?

    In a lively panel discussion for the Actuaries Institute, Taylor Fry Principal and cyber lead Win-Li Toh explored the way forward for organisations and the role actuaries have to play in shaping their cyber resilience. She was joined by Simon Mitchell, of the Australian Institute of Company Directors (AICD) and Amanda Zeller, of the Australian Securities and Investments Commission (ASIC).

    The panel sees real advantage in using traditional governance processes to strengthen cybersecurity

    The panellists covered some of the most concerning topics for boards. These ranged from the pros and cons of paying a ransom during a ransomware event to how internal cyber education fits into an information security budget, and the availability of cybersecurity professionals to service this escalating risk area.

    We outline the panel’s four key messages for organisations and their boards in building their cyber resilience:

    1. Organisations should start with their existing risk framework to understand cyber risk

    Organisations have struggled to understand and engage with cyber risk in the same way they look at other business risks. Reasons include the technical and at times jargon-heavy nature of information security technology, as well as the less clear link between cyber risk and the balance-sheet impact (compared to more familiar types of risks). AICD’s research and engagement, as part of the development of its recently released Cyber Securities Governance Principles, identified a real advantage in using and adopting traditional governance processes to engage with and tackle cyber risk. Using the existing governance and risk management approach provides a common language, helping to bridge the gap between management’s deeper understanding of the technical details and the board’s broader supervisory role.

    2. One size doesn’t fit all when it comes to comparing cyber controls

    Organisations would like to know how they are measuring up when it comes to implementing cyber controls, not to mention the money they spend on cyber resilience. But finding ‘one size fits all’ metrics is challenging due to the differences in organisations’ size, complexity and data assets, as well as the regulatory landscape.

    For example, the AICD considered including a metrics dashboard in its Cyber Securities Governance Principles, but decided there wasn’t a gold standard that would work for everyone. Similarly, while ASIC’s imminent cyber survey of regulated entities broadly aligns with the National Institute of Standards and Technology (NIST) framework for managing cybersecurity risk, it doesn’t strictly follow the NIST framework.

    3. It’s imperative to keep on top of the fast-changing regulatory and policy landscape

    In tackling cybersecurity risk, actively seeking information and staying up to date with requirements are critical. Some of the most recent policy developments with implications for organisations include:

    • The Australian Government’s 2023-2030 Australian Cyber Security Strategy Discussion Paper, released in February
    • The recently strengthened Security of Critical Infrastructure Act 2018
    • Potential changes to the Privacy Act
    • The Federal Court Judgment in the case of ASIC v RI Group Pty Ltd, which found the company breached its Australian Financial Services licence obligations to act efficiently and fairly when it failed to have adequate risk management systems to manage its cybersecurity risks.

    There is a fine balance between protecting consumer privacy and limiting organisations’ abilities to innovate and take advantage of the big data they collect. Even so, the panel believed the obligations on organisations would be increased if regulators do not see the cybersecurity uplift they are expecting from entities.

    … it is imperative organisations build a strong cyber-aware culture, and that this is led from the top.

    The panel acknowledged that growing government attention and reform in the cyber space meant organisations really needed to be proactive and build their cyber resilience now, which will put them in a good position to meet their increased obligations down the track. In addition, regardless of any policy changes, it is imperative organisations build a strong cyber-aware culture, and that this is led from the top.

    4. Actuaries have a real role to play in demystifying cyber risk for boards

    Cyber risk has been classified a ‘non-financial’ risk by boards, but the major breaches of the past year or so have shown there are direct financial implications beyond reputational risk, including customer remediation and class-action activity. A key role for actuaries going forward will be leading the modelling and estimation of cyber risk presented to boards, which would help make the risks clearer and more tangible to directors, and concentrate their focus.

    Actuaries should also consider working closely with auditors, who are increasingly including cyber risk in notes to financial statements.

    About the panellists

    • Simon Mitchell is a Senior Policy Adviser in the Education & Policy Leadership team of the AICD. He focuses on cybersecurity, financial services and the not-for-profit sector, and how policy and governance developments in these areas impact AICD members. Simon led the development in 2022 of the AICD CSCRC Cyber Security Governance Principles. Previously, Simon had an extensive career in Commonwealth regulatory agencies, including the Australian Prudential Regulatory Authority, and the Australian Competition and Consumer Commission.
    • Win-Li Toh is a Principal and cyber lead at Taylor Fry. She was one of the lead authors of the Actuaries Institute Green Paper Cyber Risk and the Role of Insurance, released in September 2022. Check out Win-Li’s profile to find out more about her work, experience and extensive industry contributions.
    • Amanda Zeller is the Senior Manager of Supervisory and Operational Resilience at ASIC. She works with a dedicated team to deliver ASIC’s Supervisory Cyber Resilience Strategy, focusing on engaging with stakeholders, building internal capabilities, benchmarking our regulated population and driving behavioural change. She works in partnership with stakeholders, including industry and government. Amanda is also Regional Commissioner, Queensland, and represents ASIC locally in this capacity.
  • What does the future of insurance look like in a warmer Aoteroa?

    Flood risk is a big issue for New Zealanders, given recent weather events and more intense climate impacts on the way. Ross Simmonds looks at the shifting landscape of flood risk and the options for decision-makers in navigating the issues of affordability and market sustainability.

    We are not yet a quarter of the way through 2023 and Aotearoa has already experienced the two largest weather events in our history. Insurance costs for the Auckland anniversary weekend floods that occurred in late January have already exceeded $1B. This event was followed by Cyclone Gabrielle, which battered large parts of the North Island in February, causing widespread flooding. The total costs of this event are expected to run into the tens of billions of dollars.

    These events have highlighted the impact climate change will have on weather-related risks for Aotearoa. As noted in the National Adaptation Plan (NAP), released by the government last year, climate change is expected to result in more frequent and extreme weather events in the future. A warmer climate means the atmosphere can hold more moisture which is predicted to increase total rainfall (total global precipitation is expected to increase by about 3% for every degree Celsius of global average warming) as well as lead to more intense rainfall hence an increasing flood risk.

    So how can we better prepare for the floods expected at our doorstep and protect New Zealanders from the devastating impacts? With flood protections tending to fall into two categories – mitigation and mop up – we break down the options and what they mean ahead for the future of insurance and the communities at risk.

    Widespread flooding was experienced across New Zealand’s North Island in February 2023

    To pool or not to pool

    In the past flood risk was largely ‘community-rated’ for home insurance in Aotearoa – that is, the cost flooding losses is spread across many homes – as flood risk information was patchy and unreliable at an individual house level. In recent years insurers have been improving their understanding of flood risks. Insurance catastrophe modellers have released a model that estimates the flood risks for all properties in Aotearoa. This enables insurers to charge premiums that more accurately reflect the flood risk for each individual property, known as ‘risk-based’ pricing. To date, most insurers are yet to implement risk-based pricing for flood risks. Given the scale of the two recent events, we can expect an increase in risk-based pricing for flood insurance in Aotearoa.

    Internationally, the introduction of risk-based pricing for flood risks has resulted in insurance becoming unaffordable for high-risk properties. In Australia and the UK, the introduction of risk-based pricing has resulted in the governments establishing insurance pools, in tandem with mitigation strategies, to provide affordable insurance.

    Would pooling work in New Zealand? The answer is not straightforward. In Aotearoa private insurers cover all hazards, including flood. As a seismically active country, the largest risk insurers face is from earthquakes. The Earthquake Commission, Toka Tū Ake EQC, is an insurance pool established in the 1940s to provide insurance coverage for seismic perils. The EQC also covers some land damage caused by floods, which is not covered by private insurers. Unlike flood risks, seismic risks are much harder to mitigate. As a result the Insurance Council of New Zealand is urging the government to focus on flood resilience measures to support properties and protect lives, as opposed to an insurance pool solution for flood risks

    The case for mitigation

    Following the recent events, the insurance industry and local government in Aotearoa have been vocal in the need for the central government to invest in flood mitigation. This will improve the ability of communities to withstand flood risks and reduce the cost of damage caused by floods. Regular investing in mitigation means costs are pre-funded prior to events occurring, rather than waiting for events to happen and incurring costs through disaster relief and recovery efforts. In a risk-based pricing insurance environment, investing in mitigation options lowers insurance premiums, as they reflect the underlying risks faced by a property.

    Managed retreat and a government buyout

    Another area discussed recently is the potential for managed retreat. This means houses in high flood-risk areas are not rebuilt in their existing location but are instead relocated to safer areas. Following the Auckland floods, there have been calls from some property owners for the government to buy them out, due to the flood risk their properties face. This approach was used in Christchurch following the 2011 earthquake, where several residential areas were declared as red zones due to the land damage. Properties in these red zones were bought out by the government. Managed retreat is not without its issues. Potentially some property owners will not want to leave but may to be forced to leave if their land is rezoned. There are also funding issues to consider, as traditional insurance does not cover land values.

    Managed retreat means that houses in high flood-risk areas are not rebuilt in their existing location but relocated to safer areas. It’s not without its issues though, as traditional insurance does not cover land values.

    Resilience-focused adaptation

    The NAP released by the government last year is part of the government’s strategy in response to the expected impacts of climate change. Two of the four priorities of the NAP are to enable better risk-informed decisions and to drive climate-resilient development in the right places. The recent events have highlighted the dangers of building in flood-prone areas. The NAP has various actions that aim to provide improved information to local councils and property owners, as well as reforming the resource management system to encourage development in areas that are less prone to climate hazards.

    The catch is that these actions will take several years to be developed. In response to the NAP, IAG NZ called for more specific, urgent and targeted steps to reduce the risk of flooding via a three-step plan:

    • A joint government and private-sector project to build common understanding of priority flood-prone communities
    • Implementing a National Policy Statement to cease development in flood-prone locations
    • Establish a national program of investment in flood.

    The last two points are mitigation actions, designed to reduce costs prior to a severe flood event occurring. This benefits communities and insurers whilst aligning with the key priorities of the government, as outlined in the NAP.

    The right path – finding balance amid complexity and uncertainty

    Due to the management and costs of flood risks being shared across multiple stakeholders, there is no clear single solution when considering how best to manage the flood risks for Aotearoa.

    The management of flood risks in Aotearoa primarily rests with local governments. Although, with the introduction of the Three Waters reform program, due to be implemented from 1 July 2024, the management of stormwater will be more centralised. When flood events occur, the costs of these events are covered by a combination of insurance companies, property owners, local and central governments.

    High premiums provide a signal for investment

    Insurers have traditionally provided financial support to their customers following an event, with no influence on how flood risks are managed, through either mitigation actions or the location of development of new properties. As insurers increase the sophistication of their premium setting for flood risks, properties that are most at risk of flood damage could see a significant increase in their premiums. In extreme cases this can result in the premiums becoming unaffordable. This risk signalling can highlight the areas where government investment in flood mitigation can have the largest impact. For example in the town of Roma in Queensland, insurance premiums increased significantly due to flood risk. Following the build of a levee, a major insurer cut property insurance premiums in Roma by 45% on average, returning these to more affordable levels.

    Risk-based pricing, relocation and buybacks

    An alternative approach to mitigation for extreme flood risks is aiding property owners to relocate to lower risk areas. Following the 2022 floods, for example, the Queensland government established a $741M Resilient Homes Fund, which includes a voluntary home buy-back program for homes that are at risk of severe and frequent flooding. As with mitigation, risk-based insurance pricing can be a driver to highlight the areas where such an approach may be needed.

    Pools as part of the solution

    Mitigation or managed retreat may not be possible for all properties that are at risk of flooding. If increasing insurance pricing sophistication results in insurance becoming unaffordable for certain properties, this can result in property values reducing as property owners will be unable to secure a mortgage. This is turn results in political pressure on government to either intervene in the insurance market, or to provide a protection pool for high-risk properties. Examples of such schemes include Flood Re in the UK and the cyclone reinsurance pool, which was established last year in Australia. Both of these schemes have been established with the aim of keeping insurance affordable.

    Insurance pools can be fraught with problems and should generally be used as a tool of last resort. Establishing a pool with no additional consideration of mitigation options does not change the underlying risks that properties face, it just changes how these risks are funded. This dampens the risk signals that private insurance provides to property owners about the risks their properties face and may result in continued poor risk management practices, such as building houses in flood-prone areas.

    Establishing pools with no additional consideration of mitigation options does not change the underlying risks that properties face, it just changes how these risks are funded.

    Insurance pools can fund more than just insurance costs, as shown by the EQC, which funds research into natural disasters and ways of reducing their impact. Pools can also be used to provide cover for uninsurable risks, such as the terrorism reinsurance pool in Australia.

    EQC levies for earthquake risk already make up a significant portion of insurance premiums in low-risk seismic areas such as Auckland. If flood risks were also included in an insurance pool, either through an expansion of the EQC or the creation of a new pool, this would further increase the size of levies relative to insurance premiums. While this reduces some of the claims risks insurers face, it also potentially reduces their profitability, which may result in some insurers exiting the domestic property market, which impacts the community.

    Last word for decision-makers – a conversation for all

    Managing flood risk is a matter of balance between risk mitigation and how to fund the costs once events have occurred. The uncertain nature of flood risks, combined with the expected increasing underlying risk due to climate change, highlight the need to continually invest in mitigation measures. Like all resources, the funds available for mitigation are finite, so thought needs to be spent on how these resources are best used. Insurers and their core function of assessing risk can assist the public sector in identifying how to get the greatest impact from mitigation costs. Bringing all stakeholders together to identify the most effective response will result in safer and more resilient communities that can adapt to the changing natural environment.

  • Climate change: key considerations for general insurers

    Australia and New Zealand have just faced another summer dominated by extreme weather events – a taste of what’s to come with a changing climate. At the same time as grappling with the immediate response to the summer’s events, insurers are also having to take a more forward-looking approach to understanding and responding to climate risk. In this article, we explore some of the key climate-related issues front of mind for Australian and New Zealand general insurers.

    Rising claim costs and reduced insurance access and affordability

    Worldwide, climate change is already increasing the risk of extreme weather events, though impacts vary by region and by type of peril. In February 2022, the Intergovernmental Panel on Climate Change (IPCC) released its alarming Sixth Assessment Report, a “dire warning” that climate change is a “grave and mounting threat to our wellbeing and a healthy planet”. In emphasising the urgent need to adapt to a warming Earth, the IPCC explains that a certain amount of climate change is ‘baked in’, so even if we immediately reduce emissions significantly (which isn’t happening), significant climate impacts will still take place.

    Climate change has, and will continue to, increase the frequency and severity of catastrophic losses in future, placing pressure on premium rates, and on insurance access and affordability in some market segments. New South Wales, in particular, has suffered through the 2021 floods, then the even worse and more extensive 2022 flood events. As the climate warms, heavy rainfall events are expected to become more intense, causing increased potential for flooding. Solutions that reduce loss of lives, livelihoods and property in a flood-prone future are complex and involve multiple parties. These include local, state and Commonwealth governments, as well as insurers, banks, builders, developers and communities. Insurance is a vital part of the solution. For example, insurance pricing, terms and conditions can provide incentives for the construction of more safely situated and resilient buildings.

    One response of the government to this issue in Australia has been the introduction of a reinsurance pool for cyclones and related flood damage, which began operations on 1 July 2022. The pool is aimed at reducing the cost of home insurance for households in cyclone-prone parts of the country.

    Insurance affordability for flood-prone areas was also on the mind of the New Zealand government well in advance of the recent events in the North Island, with New Zealand’s first national climate adaptation plan released in August 2022 listing “develop options for home flood insurance” as a key action to address inequity. This action notes that work is underway to determine the impact of risk-based pricing on insurance and explore options to support access to and affordability of flood insurance. Possible options range from doing nothing through to some form of government action, as in Australia, the UK and other countries.

    Climate risk management, adaptation and mitigation will be increasingly critical for insurers, customers and regulators into the future. This is likely to require a forward-looking approach, new capabilities and a broader range of partnerships.

    This flooded house at Windsor, Western Sydney, was a typical scene across NSW during the 2022 floods

    Standardised disclosure of climate risks

    With growing risk of climate related risks comes a growing global momentum toward standardised disclosure. The Task Force on Climate-related Financial Disclosures (TCFD) issued voluntary guidelines that 2,600 companies around the world endorsed in 2021. The recently created International Sustainability Standards Board has also released draft International Financial Reporting Standards on Climate-related Disclosures, which builds on the recommendations of the TCFD. The message for insurers is that investors and regulators will expect increasing maturity and quality of disclosures.

    In New Zealand climate related disclosures are now mandatory for accounting periods that start on or after 1 January 2023 for:

    • Large, listed companies with a market capitalisation of more than $60 million
    • Large-licensed insurers (with greater than $1 billion in total assets under management or annual gross premium income greater than $250 million)
    • Registered banks, credit unions, building societies and managers of investment schemes with more than $1 billion in assets and some Crown financial institutions.

    Given the global momentum towards standardised closures, smaller NZ insurers should get on the front foot and start thinking about how to identify, quantify and report on their climate-related risks.

    In December 2022,  the Australian Treasury released a consultation paper on Climate-related financial disclosures, which commits to introducing standardised, internationally-aligned reporting requirements for Australian businesses to make disclosures. The consultation paper proposes that large financial institutions – including insurers – should be the first cabs off the rank for mandatory disclosure requirements. The consultation period closed in February, and insurers will be keeping a keen eye on what size thresholds will be put in place to determine this initial requirement. Next steps involve further consultation on a specific design proposal which will provide more detail of the new reporting requirements, their implementation and sequencing.

    Australian insurers of all sizes should be preparing for eventual mandatory disclosures, building off the considerations they have made applying APRA’s Prudential Practice Guide CPG 299 – Climate Change Financial Risks which reflects the TCFD’s framework for considering and managing climate risks.

    A move towards net zero

    While climate-related disclosures focus on the risk that climate change poses to a business’s operations and profitability – companies, investors, regulators and other stakeholders are also looking to understand the direct impact a company’s operations and products have on the climate and environment.

    Joining the global momentum towards insurers making net-zero emissions targets, in November 2022, the Insurance Council of Australia released its Climate Change Roadmap Towards a Net-Zero and Resilient Future. This roadmap reflects the commitment of the Australian general insurance sector to achieving net-zero by 2050, and provides guidance for the Insurance Council members on the role they can play in the decarbonisation of the Australian economy.

    Further, at least 39 major insurance companies have adopted policies to no longer insure new coal projects and, in many cases, to phase out their cover for existing coal operations. In a similar way, net-zero commitments are likely to continue to impact underwriting standards in future, particularly for high-emissions industries, such as transport and agriculture.

    Opportunities for innovation

    As economies transition towards lower emissions, opportunities will emerge for industries that can demonstrate more emissions-efficient systems, or even carbon neutrality. Leading insurers will be on the lookout for ways in which they can develop new products and services, access new markets and build resilience along their supply chain. Examples already exist of banks and insurers offering better rates to businesses with lower emissions relative to their sector peers – partly because these companies are deemed better risks. An Australian insurer, for example, currently offers a low emissions car discount, where customers can save up to 25% off comprehensive car insurance if their car is recognised as having low emissions.

    How can Taylor Fry help

    With more than 20 years’ experience partnering with the insurance industry, and a team of climate risk specialists, Taylor Fry can help insurers think strategically about their response to climate change. For more information on the services we offer, visit our Climate and Sustainability page.

  • Gender bias in AI and ML – balancing the narrative

    In support of this year’s International Women’s Day theme Cracking the Code: Innovation for a gender equal future, statistics expert and Taylor Fry Director Gráinne McGuire looks at the issues impacting gender parity in this age of digital transformation. Driven by her passion and advocacy towards the ethical use of machine learning (ML), Gráinne explores the inherent gender bias in artificial intelligence (AI) and what can be done to close the gender gap.

    “Man is to computer programmer as woman is to homemaker?”. So opens a paper I came across recently. It seems to me to perfectly encapsulate the problems with gender and AI. Not that there’s anything wrong with either job – what’s wrong is the gendered association. Combine that with AI and ML being increasingly applied at scale and in a position to not only predict the future, but also create the future and reinforce biases, and we’re into Cathy O’Neil’s Weapons of Math Destruction territory.

    If data is biased, AI and ML tools amplify the biased patterns, such as woman = nurse

    We know we’ve got problems with AI and gender. AI relies on lots and lots of data. This data comes from our lived experiences and our world is biased. And AI isn’t really all that smart – it just pretends to be by being really good at finding patterns in data and using those to infer or predict things about the world. And therein lies the problem – AI suffers from bias amplification. You feed in biased data and – no surprise – the AI or ML tool finds the patterns and comes up with man = doctor, woman = nurse. Or the one above, which isn’t necessarily all that accurate.  Women were prominent in the early days of computer programming after all – with the likes of Ada Lovelace, often regarded as the first computer programmer, and American computer scientist and mathematician Grace Hopper paving the way – until men realised it was cool and important.

    Those in the know might point out that I’m referring to embedding models above and in many ways they are the larvae to ChatGPT’s butterfly (related, but utterly different and so much upstaged by ChatGPT), but ChatGPT can fall prey to the same thing (although the ChatGPT creators do appear to have deeply considered gender bias – still, it’s hard to catch everything).

    Real-life examples of gender bias in AI

    Profession gendering

    In a recent article for Fast Company, Textio cofounder Kieran Snyder asked ChatGPT to write feedback for someone in a particular job with no gender specified. Often the feedback was gender neutral but stick nurse or kindergarten teacher into the mix, or mechanic and construction worker and the she’s and he’s start to appear. You can probably guess which jobs got which pronouns. But at least doctors were always ‘they’ in their tests, so that’s something.

    Racial bias – even famous people are not immune

    Of course, it’s not just language models that suffer from bias and specifically bias against women. We see it in facial recognition software, particularly if you’re a black woman. MIT graduate, Joy Buolamwini’s research paper uncovered large gender and racial bias in automated facial analysis algorithms. Given the task of guessing the gender of a face, the systems performed substantially better on male faces than female faces, with error rates of no more than 1% for lighter-skinned men, compared with error rates of up to 34.7% for darker-skinned females. She called this phenomenon the ‘coded haze’, in which AI systems failed to correctly classify the faces even of iconic women like Oprah Winfrey, Michelle Obama and Serena Williams.

    The’ tech bro’ age

    And coming back to where we started, man is to computer programmer? Well, that’s a contributing factor to the problem. We’ve all seen pictures of the tech bros. Even those of us not in tech but using AI/ML tools in other areas often find ourselves in workplaces that have more men than women. And when people are designing algorithms and thinking about their impacts, with the best will in the world, it’s hard to be a good advocate for someone whose experience of life is very different to yours.

    Bias in facial recognition has seen algorithms fail to classify iconic women like Michelle Obama

    A matter of life and death – why we must act now

    Quite literally, this bias is killing people. Medical data, I’m looking at you. A lot of medical trials are carried out on men only because – you know – women have these awkward hormonal cycles that mess with the results. So, let’s do it on men only and get nice clean results. But, those very same hormones might mean the medicine works differently in women, or their symptoms might be different. Feed that inadequate data into an ML algorithm and the bias goes round and round. Women get inadequate health care or die.

    Invisible women – when prejudice is magnified

    This problem was explored in depth by best-selling author Caroline Criado Perez, who spent years investigating the gender data gap, and wrote the award-winning book Invisible Women. In her new podcast series, Visible women, Caroline uncovered that AI might be making healthcare worse for women because it magnifies the pre-existing bias and data gap caused by overrepresentation of men in cardiovascular research. In fact, according to research funded by the British Heart Foundation, more than 8,000 women died between 2002 and 2013 in England and Wales because they did not receive the same standard of care as men.

    The ‘strong’ vs ‘bossy’ lens

    In a wide-ranging interview with Jacqueline Nolis, a data scientist, I was struck by Nolis’s experiences with performance reviews as a transgender woman before and after transitioning. A well-established data scientist, her reviews went from being described as being “so good at always saying the truth … even in hard situations” and “thank goodness … always speaking out” to – following a job change after her transition – receiving for the first time in her life reviews like “difficult to work with”, “doesn’t know how to speak to other people”, “needs to learn … when to not say stuff”. Is it any surprise we’re losing women from the AI/ML pipeline when many of them face everyday sexism?

    Busting myths one hurdle at a time

    And that’s even assuming they get into the pipeline in the first place. They have to get past the hurdle of studying a STEM field in university, where many women come from a society that says “women just aren’t as good at maths and sciences as men”. This even became a controversial political issue in England in 2022 when the Chair of the Social Mobility Commission discussed why fewer girls take A-level physics, saying “they don’t like it, there’s a lot of hard maths in there that I think they would rather not do”. While not everyone shares that view (the Children’s Commissioner for England countered that it was more to do with the lack of female role models in STEM), it does show that this view is endemic in many societies.

    AI might be making healthcare worse for women … More than 8,000 women died between 2002 and 2013 in England and Wales because they did not receive the same standard of care as men.

    What can be done to close the gap?

    This is hard – let’s not minimise it. But hard is not an excuse for doing nothing.

    Tackle ethics together

    Ethical data standards are a place to start and there are many of these around, but they all (unsurprisingly) share common themes – recognise and manage bias, be fair, consult with those impacted by their use, consider privacy and human rights. However, it’s important to acknowledge the problem with many of these standards is that they don’t tell you how to deal with these issues.

    This point was demonstrated in feedback we received in our one-year review of the NZ algorithm charter (a “commitment by government agencies to manage their use of algorithms in a fair, ethical and transparent way”). Some of the feedback included that not all agencies had sufficient experience in measuring bias or applying human oversight, and they’d appreciate a community of practice to support compliance with the Charter as a whole. So we need to learn from one another.

    Focus on fairness

    There’s much discussion around the concept of fairness out there. But fairness is a complicated problem, with many possible and conflicting definitions. I’ve discussed fairness in the past, so I won’t repeat it here (but if you check out the article, you’ll find an example of gender bias in Swedish snowploughing, as well as a discussion around some issues of fairness).

    If you’re building a model with significant personal impacts, then you may want to consider building an interpretable model, as per another article of mine on the topic. Long story short, if you’re looking to deal with biases, then working with interpretable models can make this a lot easier, since you at least understand why your model makes the predictions that it does.

    Proactive perspective in the workplace

    Consider your staff and get a diverse range of people into the room working on these algorithms, which in turn gives you a diverse range of views on the possible consequences of using them. I’ve observed that the outspoken people on topics of fairness, bias and impact of AI are often women – Cathy O’Neil, Timnit Gebru, Cynthia Rudin, Caroline Criado Perez to name a few. Is this a coincidence?

    The thing is, gender isn’t a minority group, women and men are approximately equal in number. And if AI is biasing against close to half the population, that’s a huge problem we need to solve.

    Understand that self-promotion can be nuanced

    I’m not an HR person so it’s a bit outside my area of expertise to suggest how to do this, but there are some commonsense things we can do – like recognising that, overall, women are more likely to underestimate their abilities, men to overestimate. Or that people may come from cultures where self-promotion may be frowned upon. When making hiring or promotion decisions, we should take that into account.

    Encourage more women in STEM

    Finally (in the sense of this article, not in the sense of solutions to the bias problem!), we’ve also got to get more women into the STEM pipeline – more girls doing STEM subjects at school and university. Again, this is a huge subject in its own right, so I won’t go into details here. But I will note that one of the articles I referenced earlier, discusses recent research by the authors which found that external feedback on mathematical abilities had a significant impact on the likelihood of girls pursuing a maths-requiring STEM degree.

    Crucial flow-on effects of fixing gender bias

    Let me finish by acknowledging that gender isn’t the only thing AI has a bias problem with. Personally, being a cisgender white female puts me in a far better place than many others find themselves. Minority groups in general suffer, some more than others. But we shouldn’t fall prey to whataboutism and use that as an excuse not to fix gender bias because other groups have it worse. What’s more, a large proportion of those other groups who have it worse will be women anyway, so fixing gender bias may just help them out a little bit, too.

    The thing is, gender isn’t a minority group, women and men are approximately equal in number. And if AI is biasing against close to half the population, that’s a huge problem we need to solve.

  • Disability Royal Commission releases Taylor Fry’s report on the economic cost of violence, abuse, neglect and exploitation

    Systemic failures and maltreatment experienced by people with a disability in Australia is estimated at $46 billion annually, or $9,600 on average per person with a disability. These are among the key findings of recently published research undertaken by Taylor Fry in collaboration with The Centre for International Economics.

    The report, commissioned by the Disability Royal Commission, underscores the need for urgent action to address the many cases of violence, abuse, neglect and exploitation identified by the Royal Commission through its inquiry. It aims to contribute to a growing awareness across government and in the community of the costs of maltreatment of people with disability and the societal inequities they face daily.

    With 4.8 million people living with disability in Australia (19% of the total population), and half of this figure over the age of 65, we hope our work will assist decision-makers and policy-makers in determining allocation of resources across areas of governmental intervention to address the issue.

    Half of the 4.8 million people living with disability in Australia are over the age of 65

    How we estimate the economic costs

    Using actuarial and econometric modelling to estimate the cost, Taylor Fry and The CIE calculate the expenditure that would be avoided if violence, abuse, neglect and exploitation ceased. This is based on statistics available for the years 2021 to 2022 and take into account:

    • Systemic failures and neglect ($27.7 billion) – This includes failure by government, business and other systems to provide equal opportunity to participate in the economy and equal access to quality services. Costs include loss in labour force productivity and costs associated with lack of accessible housing.
    • Interpersonal maltreatment perpetrated by individuals ($18.3 billion) – Costs include additional hospital stays due to maltreatment and reduced quality of life.

    Key findings – Cost, experience and gaps in outcomes

    • The cost of violence, abuse, neglect and exploitation that can be reliably measured is $46 billion in 2021-22
    • 60% of people living with disability experienced maltreatment perpetrated by an adult
    • 28% of children with disability experienced significant child peer bullying in their lifetime
    • The average cost of violence, abuse, neglect and exploitation per person is double for First Nations people
    • The cost of interpersonal maltreatment is higher for women than men, while the cost of systemic failures is higher for men.

    Gaps in outcomes, which are indicative costs from other forms of over-representation in poor-outcome areas and under-representation in financial and social opportunities are valued at $28.8 billion in 2021-22. Specifically, these include:

    • Finding it more difficult to secure employment when a person with disability rates themselves as ‘able to work’
    • Poor health outcomes from higher rates of obesity, lower rates of physical activity and higher blood pressure compared to other Australians
    • Higher rates of homelessness, involvement with the justice system and loneliness.

    We hope our findings will assist decision-makers to allocate resources and inform interventions

    The report also states that costs would likely be higher if more data was available. Substantial savings can be made over time if governments and society are successful in addressing maltreatment, including issues of systemic failure and neglect.

  • A life in the career of an actuary

    Finding your way in any profession can be filled with adventure, challenge and plenty of life lessons – while figuring out the right mix of passion and purpose is perhaps the biggest skill of all. For actuary Anh Vu, balance is key, as she charts a path to bring the best of academia and industry together for mutual benefit and meaningful change in people’s lives. She takes us on her journey of actuarial discovery.

    As an undergrad, I often wondered what I’d do when I became an actuary. Mostly, I imagined solving financial problems in an insurance company or a bank. Like many actuaries, I’ve always had a passion for maths, business and economics. At the very least, I hoped the distributions and mathematical models I managed to memorise for exams would one day be useful in my future job.

    I wasn’t wrong, but I definitely underestimated the potential!

    Anh shares her journey and learnings across both traditional and non-traditional actuarial paths

    Free-style innovation

    After my commerce degree and honours in actuarial studies, I was keen to stay in academia and continue with a PhD. I enjoyed research and the idea of generating innovation completely free style – where you pick topics you love and take ownership of your work – and the flexible hours appealed to me as well. I chose a joint program in actuarial studies at UNSW and applied mathematics at Universite de Montreal, which meant living in Canada for a whole year.

    My academic journey helped me learn that being able to understand and think critically is vital towards developing a diverse skillset as an actuary. For example, I explored advanced modelling techniques that had applications in robotics engineering to solve actuarial problems. The work was inspiring and made me realise we don’t need to be restricted by what we’re taught in university when we’re choosing statistical and mathematical tools. I’ll never forget the experience.

    Theory meets practice

    By the end of my PhD, I was ready for a new challenge. Research was on the theoretical side, and I wanted to know the practical side of the profession. It struck me that there’s often a gap between the theory and what happens in real life, and I wanted to help close that gap. It made sense to me that academia and industry could benefit more from each other. These days, I’m happy to be working in a consulting environment. The work is project based, so it has given me plenty of opportunity to immerse myself in a variety of projects and apply the theory to solve real-world problems in different areas. I never get bored!

    Finding meaning

    It helps that I’m part of a team with a reputation for pioneering work in the government sector, using actuarial techniques to support policy design and improve people’s lives. It’s fascinating but also meaningful. Luckily, we’re able to work flexible hours, which was something I appreciated through my PhD and was keen to continue in my job to support wellbeing, and maintain a healthy balance of passion and perspective.

    Pathways to better outcomes

    In my working life so far, I’ve had a taste of the traditional and non-traditional actuarial paths. The traditional path involved helping insurers with their pricing and liability valuations. The non-traditional path was using advanced modelling techniques to support government in making data-informed decisions and improve social outcomes.

    Through these roles, I’ve learned many important lessons and there have been lots of highlights.

    When it all makes sense

    In the traditional environment, it became clear that actuarial work is a lot more than just applying some statistical formulas – it’s about making sense of the numbers and methods you use with the business context firmly in your mind. Doing technical work is important, but it’s equally important to be able to communicate your technical work and results to different stakeholders – especially those decision makers who may not deeply understand the technical elements and want to know the impact for their business in straightforward terms.

    It was also handy to have work experience in the traditional areas as I was studying for my actuarial fellowship with the Actuaries Institute – and it definitely helped me pass the exams!

    Sometimes, I think actuaries are like superheroes and our actuarial skillset is our superpower

    A versatile, evolving skillset

    The non-traditional work in the government sector showed me the versatility and evolving nature of the actuarial skillset – the learning never stops! I’ve always been surprised by how much I can do with it, too. It’s extremely rewarding to know that I can use my actuarial skillset towards meaningful change in people’s lives.

    Not all superheroes have capes

    Sometimes, I think actuaries are like superheroes and our actuarial skillset is our superpower in the way we can contribute to social good. The range of areas our work applies and can make a difference is so diverse as well, from banking to government, marketing, climate change, and many more.

    And when our capes come off? Equally important in the life of an actuary is the fun and social aspect of our profession. I’m fortunate to have been surrounded throughout my career by like-minded and compassionate people who inspire me – which has always made and I’m sure will continue to make the journey truly enjoyable.

  • Game on – winning formula for an actuarial Excel whizz

    By day, Taylor Fry Director Andrew Ngai is hard at work for his clients. By night, his knack for numbers often plays out very differently. We caught up with him to hear how his passion for spreadsheets became a surprising TV sporting sensation and what it takes to be an Excel gaming champ.

    You recently won the Microsoft Excel World Championship (MEWC) – how did you first become interested and what appeals to you most about taking part?

    Excel is part of my day-to-day work as an actuary and I thought I’d have fun with it when a few years ago I discovered a competition called the ModelOff Financial Modelling World Championships. From there, I found out about MEWC and was keen to enter for its non-financial focus. I find these kinds of problems particularly enjoyable to solve – the scenarios are imaginative and there’s lots of opportunity to think creatively. It’s also been nice to meet fellow Excel enthusiasts from all over the world through these competitions.

    King-size success: Andrew adds the MEWC crown to his All-Star Battle win earlier this year

    How can you top this performance next year? Any special rival you’re watching out for?

    I’ll have my work cut out for me! There are so many excellent Excel modellers involved and the competition is bound to become fiercer as more people find out about it, especially with the huge media coverage and viral response to the Excel Esports All-Star Battle I won earlier in the year. Quite a few of the players have also been competing in other financial modelling competitions for years, sharpening their skills. As for a special rival, I’d say Diarmuid Early – he’s really fast at problem solving and has deep Excel knowledge. We faced off in the All-Star Battle, and although I managed to edge him out in the final, he finished the semi-final question a lot faster than I did.

    How did you react when you first discovered the competitions would be broadcast on ESPN – the world’s leading sports entertainment broadcaster, with an average day-viewership of 812,000 people? Who did you imagine your audience would be?

    It caught me by surprise – I never imagined I’d be on TV, let alone for an Excel competition. I don’t think it sunk in until after the broadcast of the All-Star Battle. With all the publicity that received, I ended up being interviewed on Channel 7’s Sunrise program. Breakfast television definitely wasn’t part of the original career plan!

    For MEWC, the organisers put a lot of effort into making the competition interesting for a broader audience, and Excel is a software that most people will have on their computers even if they don’t use it much. So in terms of the audience, while I suspected it would appeal mostly to people familiar with Excel in their day jobs, I thought it would also attract some other curious spectators.

    With all the publicity … I ended up being interviewed on Channel 7’s Sunrise program. Breakfast television definitely wasn’t part of the original career plan!

    How relevant are the Excel skills you need for the competition to your day job?

    The skills overlap a lot, and I think it goes both ways – my day job helping me in the competition and vice versa. For example, I started using more dynamic arrays in the competition, which help perform some tasks much more efficiently (e.g. sorting or filtering for unique values), and brought those over to some of my work. Conversely, I’ve been using formulae like INDEX and MATCH for many years, for example to efficiently look up values within tables of data. These are also core to solving many MEWC or financial modelling problems.

    How was the experience for you this time with the live broadcast? Did it affect your focus?

    The live broadcast does make me feel a bit more pressure knowing people are watching, especially this year, going in as one of the favourites to win. Luckily, I managed to focus reasonably well once we started solving the problems. It helped to get a few minutes of reading and thinking time before the timer started.

    How did you feel the moment you won?

    I was delighted, but also exhausted after two hours of intense problem solving in the middle of the night! With the competition becoming stronger, I was happy to defend my title this year. I was also thankful for some good fortune along the way, especially in the semi-finals when I made a last-minute lucky guess to just beat my opponent’s score.

    Peak performance: Andrew defended his 2021 MEWC title with a mix of focused prep and a little luck

    How do you prepare to stay in mental and physical shape?

    For me, staying in mental shape is a balance between practising past problems but also not solving so many that I tire of it all and burn out. Physically, it was about doing what I can to be as awake as possible at 4am. A shower before the competition started was a must!

    How do you juggle work and the competition? How much sleep do you manage over the duration?

    I guess it helps that my work involves Excel a lot, so in that sense doing work is also helping me to prepare. In the week leading up to the finals, I sacrificed a bit of my social life (willingly!) and spent most of my nights doing or reviewing past problems. Sleep was mainly a challenge on the last night, with the finals being held at 4am in my time zone – I can’t say I had quality sleep the night before.

    Is it a young person’s game or can you see yourself still competing years from now?

    While younger people may be a bit quicker with using the keyboard and mouse, being successful in the competition is more about having powerful problem-solving skills, and there are plenty of strong participants who are older than me.

    What helps you perform at your best?

    Training and preparation help me a lot, including familiarising myself with the types of problems I’ll likely encounter. I’m able to concentrate pretty well once I get started and I make sure there are no distractions. Ample reading time to understand the question is important, too. Some questions involve more complicated games where it takes longer to understand the rules. I usually struggle with those and rush into building the solution too quickly because I haven’t had the time to think about the solution.

    How do your friends and family view your participation in the competition?

    My friends and family are all very supportive and encouraging. I know my parents watched the whole thing live, but most others only watched afterwards given it was well before dawn our time!

  • RADAR New Zealand Snapshot

    The past year has presented plenty of challenge for the New Zealand insurance sector, with events unbound by global borders shaping the landscape. Climate change, rising inflation and cyber perils are among the top priorities around the world and impacting insurers locally.

    Download our report

    to discover the effects of what’s been and how to prepare for what lies ahead

    In RADAR New Zealand Snapshot, we use the latest data and our market expertise to shed light on the New Zealand general insurance landscape, and assess the patterns, drivers and emerging trends across the industry. Our overview highlights key market movements and explains what they mean for insurers, from the impacts of a global rise in natural disasters and inflation to climate adaptation, regulatory change and cyber risk.

    Underpinned by ICNZ and RBNZ figures, as well as research by The Task Force on Climate-related Financial Disclosures and recent Actuaries Institute green paper, we offer insurance insights to help decision-makers build resilience and sustainability for Aotearoa into 2023 and beyond.

  • Accuracy and causality the hot topics in data science at SIGKDD 2022

    The international data science community gathered recently in Washington DC for the first in-person conference of its kind since 2019, sharing insights on knowledge discovery and data mining. Taylor Fry’s Tom Moulder was there for the 28th Association for Computing Machinery SIGKDD event, and reveals two big themes – why model accuracy is no longer enough and the growing embrace of causal relationships.

    KDD styles itself as the world’s top interdisciplinary data science conference. It’s a fair assessment, given KDD has introduced some of the most influential developments in machine learning (ML) – for example, the now ubiquitous XGBoost algorithm.

    This year featured speakers from Google, DeepMind, Amazon, DARPA, Microsoft, Meta, LinkedIn and the world’s top universities. Talks spanned a range of applications, including recommender systems, social networks, health, autonomous vehicles, finance, social impact and ecommerce.

    Despite this breadth, there were a couple of key challenges that featured consistently:

    • Beyond accuracy – How can we build ML applications that are not just accurate, but also fair, unbiased, robust, privacy preserving and transparent?
    • Capturing causality – How can we build models to understand causal relationships, and empower us to analyse the best decisions to make, rather than just predict outcomes based on history?

    Causation analysis has relied on randomised experiments, often wildly expensive or impossible to run

    Finding value beyond accuracy

    As the ML field matures, models are increasingly required to be more than just accurate. They should also be transparent, explainable, fair, privacy preserving and robust to exploitation.

    This is echoed in 2021 comments from European Commission Executive Vice President Margrethe Vestager, concerning European Union rules on artificial intelligence: “… trust is a must, not a nice-to-have”.

    These challenges have been a focus of ML researchers and practitioners in recent years, appearing in KDD talks and workshops across almost every application and industry. The issues are particularly relevant for actuaries in our approaches to modelling and problem solving.

    The argument for a human-centric approach

    At the conference, Krishnaram Kenthapadi (Fiddler AI), Hima Lakkaraju (Harvard), Pradeep Natarajan (Amazon), and Mehrnoosh Sameki (Microsoft) presented a workshop on model monitoring, in a world where achieving model accuracy is no longer enough – rather, it’s just one of numerous considerations with potentially serious consequences. They make the case that continuous monitoring is critically important, and should involve evaluating biases, (un)fairness and model robustness rigorously. They argue for ‘human-centric’ modelling, whereby monitoring provides analyses to guide model review, while preserving human decision-making around questions such as when to review and update models.

    Tackling bias

    A full day workshop on Responsible Recommendations continued this theme, raising issues such as fairness in algorithms recommending job applicants for open positions. James Caverlee (Texas A&M) discussed the risk of ‘mainstream bias’, whereby recommendation engines may be biased towards performing well for mainstream users, but more poorly for niche users or minority groups. He went on to propose data augmentation and model tuning methods to tackle these biases.

    Glassbox modelling – Powerful and explainable

    Rich Caruana and Harsha Nori, of Microsoft Research, further emphasised the importance of metrics beyond pure accuracy, stressing model explainability and privacy in their new ‘glassbox’ modelling approach. This capitalises on the power of ML methods, while retaining a simple and explainable final model structure.

    They demonstrated the advantages of their approach in a case study predicting risk of sepsis in pediatric hospital patients. Their explainable modelling structure highlighted the potential for patient risk scores to vary unexpectedly – likely due to data biases caused by treatment processes. Physicians were then able to manually edit the model to better reflect their knowledge and remove the unexpected effects.

    The ability to understand and meaningfully engage with the model enabled physicians to improve and trust it – a crucial factor in applying models successfully.

    Applying models well requires understanding and meaningful engagement to improve and trust them

    Correlation vs causation

    Anyone who has spoken with a statistician will have heard the phrase ‘correlation does not equal causation’ ad nauseum. However, many of the highest value data science applications are questions of causation. Should I prescribe a patient medicine A or B? Charge price X or Y for my product? Invest in marketing campaign C or D?

    Traditional analytics and ML methods aren’t designed with these questions in mind, and often fall short. They model correlations between variables, but don’t directly estimate causal relationships. Analysis of causation has relied on randomised experiments, which, while effective, are often prohibitively expensive or impossible to run.

    Emphasis on causal relationships

    In the past decade, developing methods to understand causal relationships has been a key focus of academia and industry. In 2018, world-renowned computer scientist Judea Pearl brought the idea of causality to a more general audience, with his widely popular The Book of Why. Most recently, the subject even featured in the 2021 Nobel Prize in Economics, which was awarded to Angrist and Imbens for their contributions to analysis of causal relationships.

    In keeping with this trend, causal modelling was a key focus of KDD2022.

    Making a difference for business

    A team from Vaiani Systems discussed the benefit of causal approaches in business, using churn modelling as an example. They grouped data science tasks into three levels: Descriptive (what happened), Predictive (what will happen) and Prescriptive (what to do). While a predictive model can identify customers likely to churn, a prescriptive one distinguishes between those who will always churn (lost causes) vs those who can be influenced to stay (persuadables). Separating these groups allows marketing budgets to be focused on persuadable customers and improve overall retention.

    They cite further examples from LinkedIn, Uber, Netflix and Amazon, where adopting causal approaches has made a real difference for the business and its customers. They also stressed the importance of developing offline evaluation and simulation infrastructure to support development and testing of causal analyses.

    The best decisions don’t necessarily come from models with high traditional predictive accuracy

    What makes a good decision?

    In public health, Professor Milind Tambe shared an example from his work as Director in at Google Research’s AI for Social Good. His team uses models to allocate resources to call centres, which provide health advice and reminders to pregnant women in India. While the team have seen good results so far, they have observed that models with high traditional predictive accuracy metrics do not necessarily result in better decisions in the field. He makes the case that this is because standard ML approaches focus on accuracy rather than decision quality. His team is now working on approaches to directly elevate decision quality – an approach conceptually aligned with causal modelling.

    Improving evaluation tools

    Tambe’s example is one of many where the standard model building and evaluation tools used by data scientists often don’t perform well on causal problems. The field is quickly developing new tools to fill this gap, with examples from this year’s KDD including new feature selection approaches for causal problems, and tutorials highlighting recent developments in multiple causal analysis packages.

    This ties closely to the idea that models should be evaluated on more than just accuracy – the most accurate predictive model is not always the best one and does not always translate to high-quality decisions. In some cases, standard predictive models and evaluation metrics can be misleading to the point where more accurate models can result in worse decisions!

    Where to from here?

    Overall, KDD provides evidence of increasing recognition of the need for causal analysis, which improves decision-making across industries. While the range of tools available to meet this need is maturing, the problem remains challenging and relies on certain types of historical data collection. Where possible, experimentation is important to test whether predicted benefits are true in practice, and support future development of more causally focused models.

    Ultimately, models should be evaluated with a holistic view of the problem they are trying to address. In practice, this means considering fairness, bias, privacy and robustness – beyond accuracy alone. Supported by interpretable model structures, this will help gain greater trust from end users and open the door to expert input on models.