Data Scientist

Role:

Lead ML Developer (NLP)

Job Summary:

This job requires candidate to lead the design and development work with Large Language model and application of Large Language models for different use cases. As an expert in using Generative AI, particularly LLM models, and proficient in Python programming, he/she will be responsible for developing and deploying a system that can convert natural language queries to handle the use cases like building SQL query statements for searching tables in DB or searching the documents for Q&A. This role requires a deep understanding of natural language processing (NLP) techniques, database management systems, SQL Query optimization, and passion for building intelligent systems.

Responsibilities:

• Develop and implement a robust system for converting natural language queries with help of prompt engineering into useful format, which can be further programmatically used for search a set of SQL tables or set of documents.
• Development and application of word embedding techniques (Woprd2vec, Transformer model based encoders, LLM based encoder, BERT based encoders etc)
• Fine Tuning of Large Language Models with custom data sets.
• Utilize Generative AI techniques, leveraging Large Language models, to accurately interpret and convert natural language queries into SQL statements.
• Use of LLMs for the NLP state of art techniques like Text Classification, NER, Keyword Extraction, Text-to-SQL conversion.
• Check the feasibility of use cases based on Large Language Models, specifically in area chatbot for sales and finance problems.
• Develop mechanism to summarize SQL tables into user-level summary text, providing concise and meaningful insights from the data retrieved.
• Design and build scalable architecture that can handle a large volume of queries efficiently, ensuring high performance and minimal latency.
• Collaborate with cross-functional teams, including data scientists, software engineers, and database administrators, to understand requirements and integrate the solution into existing systems.
• Conduct thorough research and stay up to date with the latest advancements in NLP, machine learning, and Generative AI to continuously improve the system’s accuracy and efficiency.
• Optimize and fine-tune SQL queries to ensure efficient data retrieval, taking into account query execution plans, indexes, and query performance optimization techniques.
• Develop testing frameworks and conduct rigorous testing to validate the system’s accuracy, reliability, and scalability.
• Document the system architecture, design decisions, and codebase to facilitate future maintenance and enhancements.
  
  

Skills

MUST

• Strong experience in architecting and development of ML, NLP based projects is a must.
• Strong track record of ML led solution development from scratch.
• Strong proficiency in Python Programming (very strong Python credentials only apply) and experience with relevant libraries and frameworks for NLP such as NLTK, spaCy, Hugging Face transformers.
• In-depth knowledge and experience in using Generative AI techniques for NLP tasks, preferable with Large Language models (e.g., GPT-3, GPT-4), GCP PALM models (code bison, text bison) or Hugging Face models.
• Good exposure to word embedding techniques. (BERT, Word2Vec, LLM based encoders etc.)
• Experience with machine learning frameworks (e.g., TensorFlow, PyTorch) and deep learning architectures.
• Hands on experience with Langchain framework.
• Solid understanding of SQL and experience working with popular relational database management systems.
• Proficiency in writing both simple standard SQL queries and complex joining queries to fetch data from a database.
• Experience with query optimization techniques and understanding of indexes, execution plans, and performance tuning.
• Familiarity with cloud-based data warehousing platforms.
• Strong problem-solving skills and ability to translate business requirements into technical solutions.
• Excellent communication skills to collaborate effectively with multidisciplinary teams.
• Ability to work independently, manage priorities, and deliver high-quality results within project timelines.
• Strong attention to detail and a commitment to producing clean, well-documented code.
  
  

Natural Language Processing - NLP