LLM Evaluation Engineer

16 hours ago

Belgrade Serbia Remote 4Create Software Full time €40,000 - €80,000 per year

We are committed to improving patient outcomes and advancing industry standards through innovative AI-powered solutions. Join our team and help shape the future of biopharma technology.

About us:

  • We are a team of experts creating software for managing different phases of clinical trials.
  • We create end-to-end products that enable our clients to discover new treatments and therapies.
  • Our biggest strength is our team of talented, passionate and devoted people.
  • We are driven by enthusiasm for working on cutting edge solutions and problems.

Here's what you will be doing:

  • Design, develop, and maintain automated test frameworks for LLM and AI-powered applications, primarily using Python.
  • Collaborate with DevOps, AI engineers, and developers to understand model requirements, data pipelines, and application workflows.
  • Develop and execute test plans and test cases for GenAI features, APIs, and cloud-based services.
  • Implement automated testing for model evaluation, data validation, and API endpoints using modern test automation tools (e.g., Pytest, Robot Framework).
  • Integrate automated tests into CI/CD pipelines in GitLab.
  • Monitor, analyze, and report on test results, identifying root causes of failures and ensuring high quality releases.
  • Ensure security, scalability, and performance testing of AI models and cloud applications.
  • Contribute to best practices in test automation, code quality, and DevOps within the GenAI team.

About you: 

  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • Proven experience in software development and test automation with Python.
  • Strong understanding of cloud-based services, with experience in testing GenAI features and APIs.
  • Experience with CI/CD pipelines (GitLab CI/CD experience is a plus).
  • Strong problem-solving, communication, and collaboration skills.
  • Experience in performance, scalability, and security testing of cloud applications.

Nice to have:

  • Experience leading test strategy design and mentoring team members.
  • Familiarity with DevOps practices and a commitment to continuous improvement in testing strategies.
  • Ability to stay current with industry trends and advancements in AI and cloud technologies.
  • Interest in staying current with AI/cloud advancements and bringing new ideas into practice.

Company benefits:

  • Fully flexible working hours
  • 100% remote working environment
  • Private health insurance (for domestic citizens only)
  • 20 PTO days + 2 Additional well-being days
  • Home office budget + work equipment (for domestic citizens only)

If you're excited by the challenge of developing cutting-edge software and working with a team that explores new technological frontiers, join us in our journey.

By submitting your application, you allow 4Create to handle and store your data.

  • Machine Learning Engineer

    1 week ago

    Remote - Serbia Rasa Full time €40,000 - €80,000 per year

    Your turn to start the conversation.Write the future at Rasa. Conversation. It's the thread between our product and our people. The tool that enables us to forge relationships through compassion and expertise. To find the connection between our differences. It keeps us close together across borders and backgrounds and helps us create our shared vision. ...

  • AI/ML TechLead

    16 hours ago

    Yerevan, Armenia / Serbia / Poland / Romania / Spain Provectus Full time €60,000 - €120,000 per year

    About project Provectus helps companies adopt ML/AI to transform the ways they operate, compete, and drive value. The focus of the company is on building ML Infrastructure to drive end-to-end AI transformations, assisting businesses in adopting the right AI use cases, and scaling their AI initiatives organization-wide in such industries as Healthcare & Life...

  • Machine Learning Engineer

    16 hours ago

    Remote - Serbia Rasa Full time €30,000 - €60,000 per year

    Your turn to start the conversation.Write the future at Rasa.Conversation. It's the thread between our product and our people. The tool that enables us to forge relationships through compassion and expertise. To find the connection between our differences. It keeps us close together across borders and backgrounds and helps us create our shared vision.Rasa...

  • AI/ML TechLead

    16 hours ago

    Poland / Serbia / Slovakia / Romania / Spain / Armenia Provectus Full time €80,000 - €120,000 per year

    Provectus helps companies adopt ML/AI to transform the ways they operate, compete, and drive value. The focus of the company is on building ML Infrastructure to drive end-to-end AI transformations, assisting businesses in adopting the right AI use cases, and scaling their AI initiatives organization-wide in such industries as Healthcare & Life Sciences,...

  • Lead AI Solution Engineer

    16 hours ago

    Novi Sad, , Serbia IWConnect Full time €60,000 - €120,000 per year

    We are looking for an AI Solution Engineer to join and lead our AI Solution Delivery Team. In this role, you will design and develop scalable applications, integrate AI models into real-world solutions, and optimize LLMs, vector databases, and AI-powered applications.  Our ideal candidate has a strong consultancy mindset, strong software development...

  • AI Engineer

    16 hours ago

    Belgrade, Serbia, Tyllo Full time €60,000 - €120,000 per year

    We're looking for a talented and driven AI Engineer to join our team and help shape innovative solutions that make a meaningful impact. In this role, you'll collaborate closely with data scientists, engineers, and product teams to design, build, and deploy scalable AI models and systems.What will you do?Design, develop, and deploy AI models;Develop and...

  • Applied AI Research Lead

    2 weeks ago

    Belgrade, Central Serbia Toloka Full time €40,000 - €80,000 per year

    Company IntroAtToloka AIwe create data that powers leading GenAI models and innovations. We work with frontier labs, big tech, renowned AI startups, enterprises and non-profit research organizations worldwide. We use a combination of Experts + Crowd + Tech Platform to teach AI models to reason and evaluate their efficacy and safety. We have experts in more...

  • Senior Machine Learning Engineer

    1 week ago

    Belgrade, Central Serbia Perplexity Full time $150,000 - $200,000 per year

    Perplexity is an AI-powered answer engine founded in December 2022 and growing rapidly as one of the world's leading AI platforms. Perplexity has raised over $1B in venture investment from some of the world's most visionary and successful leaders, including Elad Gil, Daniel Gross, Jeff Bezos, Accel, IVP, NEA, NVIDIA, Samsung, and many more. Our objective is...

  • AI Developer

    16 hours ago

    Belgrade, Central Serbia Genestack Full time €80,000 - €120,000 per year

    At Genestack we are tackling the underlying computational and scientific challenges of bioinformatics in order to provide researchers with software tools that will streamline the discovery process and drive forward precision medicine, drug development, and bioinformatics research.We are seeking a senior or lead-level AI Developer with a strong focus on LLMs...

  • Shape the Future of AI — Serbian Talent Hub

    16 hours ago

    Serbia / Remote, Worldwide / Remote - Europe Welocalize Full time €15,000 - €30,000 per year

    Welo Data, part of Welocalize, is a global AI data company with 500,000+ contributors delivering high-quality, ethical data to train the world's most advanced AI systems. We're building smarter, more human AI with a diverse community in 100+ countries. At Welo Data, Limitless AI. Limitless You. isn't just a slogan—it's our promise. We build smarter AI...