Initial steps in active learning with NEXT

26 Oct 2018

Active learning is a semi-supervised machine learning technique where the instances are chosen depending on gathered data. NEXT is a cloud based machine learning system for active learning. It provides a lot of built-in features such as the UI for crowsdourced experiments, an HTTP API, experiment monitoring dashboards and stats for the experiments carried out. The researcher/developer must implement the machine learning algorithm and recruit participants for adaptive data collection - the rest is handled by NEXT.

Built-in examples in NEXT

Cardinal bandits

Cardinal bandits ask participants to rate 1 object. For example, if we think about the New Yorker Caption Contest example in NEXT, an example of this is provided when we rate the caption for an image as funny, not funny or somewhat funny.

Dueling bandits

Dueling bandits ask participants to rate 1 of 2 objects. It is a variation of the multi-armed bandit problem. If we think about another version of the New Yorker Caption Contest example in NEXT, an example of this is provided when there are 2 captions for an image and we rate 1 of the captions as funnier than the other.

Triplets

This displays 3 objects and asks for triplet responses. So an example of this is provided in NEXT via displaying 3 images A,B and C and asking the user to rate one of 2 images B and C as more similar to image A.

How do I run a built-in active learning experiment locally with NEXT?

  • Clone the “NEXT” repository
  • In a separate folder inside /next/local, copy your experiment image, populate the init.yaml file to pass in experiment algorithms and parameters, pass data as a zip folder
  • Run ./docker_up.sh in /local
  • Run python -m SimpleHTTPServer 8999 in the experiment folder (eg /local/cartoon_dueling)
  • Either manually set targets in init.yaml file and run ‘python launch.py cartoon_dueling/init.yaml’ inside /local, or go to localhost:8000, select launch experiment, then upload init.yaml as arguments file and captions zip file as targets file, and click launch

Here are some screenshots of a sample experiment for dueling bandits (for the New Yorker Caption Contest)

Launching an experiment

Launching an experiment

Experiment query page

Experiment query page

Experiment dashboard

Experiment Dashboard Experiment Dashboard

Very briefly, that sums up some of my initial attempts at getting to know active learning through NEXT. There’s certainly a lot more to explore.

For more information

Web Statistics