AI Careers Opening Up to Roles That Surround Core Technical Team
The growing readiness of AI tech is changing the narrative; LLMs, for example, can be a foundation to support multiple AI use cases
By John P. Desmond, Editor, AI in Busines
Most technology applications can be understood by deconstructing the individual building blocks of code, and debugging the software if something is wrong, stepping through the code line by line if necessary, to find out where the problem is. AI is different.
The latest large language models, notably GPT-3 from OpenAI, have outcomes that are less than predictable. GPT-3 was trained on billions of pages of text scraped from the internet and contains hundreds of billions of parameters.
“And although we can’t predict the exact outcome, we can train them to do what we want. This is leading to hugely exciting use cases,” stated Penny Li, SVP, VLSI at SambaNova Systems, author of a recent account in Open Access Government
The academic researchers who brought AI to where it is today are now taking on a more “custodial” role, giving away to a range of “individuals from all walks of life with different perspectives. “They are greatly needed to support technology’s growing role in all fields,” she stated.
The growing availability and readiness of AI technology is changing the narrative, with certain forms of AI able to support multiple use cases in business. Large-language models (LLMs), for example, provide “a foundational AI model that can be used in a variety of applications, demonstrating the departure from the often singular uses of academic AI,” she stated.
As a result, more workers familiar with AI are needed, but not only highly-trained data scientists and algorithm tweakers. AI education offerings are increasing, with 229 undergraduate university courses available at the time of the writing (October 2022) in the UK alone, compared to 196 courses on software engineering.
This makes for a good opportunity for recent college graduates with some AI course work. “Entry-level candidates with minimal experience have inherently found the technology more accessible than previous generations,” Li stated, enabling them to contribute fresh thinking and new solutions incorporating AI.
For example, she sees that those with a background in analytics are valuable for machine learning translation roles, since they are able to make sense of an algorithm’s patterns in a business context. And those with communications skills can complement the AI in cybersecurity technology, able to convey the severity and risk of a security incident throughout an organization.
“The start of the AI revolution is upon us,” Li stated. It is no longer dark magic.”
Experience of AI At Work Sheds Light on Opportunities
Examples of what people are doing at work with AI can shed light on the opportunities. A recent account in KDnuggets provided examples of how machine learning is being applied in enterprises.
Connie Yee is a senior data scientist at Bloomberg, whose role is to build ML solutions that help improve the company’s data and analytics, supporting over 300 billion requests a day. “We have a lot of opportunities to model non-traditional ML problems [using] standard ML and data science algorithms,” she stated. For example, her team works to ensure that code that calls out to services is modularized, which does not present as a standard data problem. “But by adopting a graphical approach and representing code dependencies as a graph, we gained actionable insights into our application code using graph theory and embeddings” she stated.
Moody Hadi is a group manager working on financial engineering at S&P Global Market Intelligence. He works on one of the most difficult challenges for AI in financial service - explainability. His team has used structural models that help analysts to understand dependencies among multiple independent and dependent factors. He says explaining these dependencies to end users is challenging. Other challenges he engages in include hyperparameter tuning, bias-variance tradeoff, and collecting high-quality, balanced data for training AI models.
Sha Edathumparampil, Chief Data Officer at Baptist Health South Florida, sees that ML will touch nearly all aspects of the enterprise in the coming years. "AutoML frameworks, as well as ML as-a-service type solutions, are going to mature to the point where they are the first option for anyone looking to solve a business problem with ML solutions," Sha stated.
Lots of work to be done. Online training continues to be an option. Simplilearn, founded in 2010 and based in San Francisco, and Bangalore, India, is an online bootcamp for digital economy skills training. The company has trained over three million professionals, has over 2,000 qualified trainers and offers over 400 courses.
Simplilearn has recently published an AI Career Guide: A Playbook To Become an AI Expert, with a guide to where the hottest opportunities are in AI, a learning path and how to get started. The company recently launched a Post Graduate Program in AI and Machine Learning with Purdue University, in collaboration with IBM, and an MS In Artificial Intelligence in collaboration with the International University of Applied Sciences, based in Erfurt, Germany.
Data Scientist or AI Engineer? Considerations for Each Path
For those interested in pursuing a career in AI but unsure where to focus, a recent account in mint offered a partial guide by comparing the role of data scientists to that of AI engineers.
A data scientist primarily concerns himself or herself with extracting insights from data, for data-driven decisions, using a combination of fields including statistics, mathematics, programming and machine learning. It does not require core subject knowledge or the skills required to create ML algorithms.
Tasks typically assigned to a data scientist include:
Understand the business problem, think about a data-driven solution;
Gather relevant data, varying from tabular to unstructured;
Create data pipelines that prepare, clean and transform the data for further analysis;
Use statistical analysis to understand the data and extract insights;
Use predictive techniques to create a machine learning model to solve the business problem;
Improve and optimize the ML model with hyperparameters for fine-tuning
Collaborate with business teams, data analysts and other stakeholders to improve the model;
Support the deployment of the model into production;
Maintain effective communication with stakeholders
Tasks typically assigned to AI engineers include:
Understand the business problem and develop an AI use case;
Evaluate the data requirements for the use case;
Gather data and perform data labeling as required;
Create a data pipeline by integrated needed data sources;
Design the overall architecture needed to support deployment of the AI model
Develop and API to provide a scalable solution to integrate AI models into selected applications;
Develop the AI model, which could incorporate AI techniques such as deep learning algorithms and natural language processing;
Train the AI models and fine tune to the requirements;
Build a stable deployment environment;
Test and deploy the AI model;
Maintain all the code and related artifacts through version control repositories
The choice between data scientist and AI engineer boils down to individual preference based on skills, ambitions and interest. Both jobs pay well.
Read the source articles and information at Open Access Government, in KDnuggets and in mint.
(Write to the editor here.)