Stephan Seeger

GNSS & Machine Learning Engineer

Category: NLP (Page 3 of 5)

OpenAI releases GPT-4

March 14, 2023 / / 0 Comments

OpenAI released GPT-4 within ChatGPT on March 14, 2023, described in detail in a 98-pages paper (summarized on youtube).

  • Available to ChatGPT-Plus subscribers (currently with a cap that is changing over time, e.g. 100 messages every 4 hours, or 25 messages every 3 hours).
  • Still based on training data that cuts off Sept 2021.
  • It still does not learn from its experience.
  • Still no internet access.
  • The training was already finalized in Aug 2022.
  • Fine-tuned via RLHF (Reinforcement Learning with Human Feedback).
  • API waitlist is open (so no API access yet for everyone)
  • API prices (for comparison: GPT-3.5-turbo $0.002 per 1k tokens):
    • gpt-4: 8K context window (about 13 pages of text) will cost $0.03 per 1K prompt tokens and $0.06 per 1K completion tokens.
    • gpt-4-32k: 32K context window (about 52 pages of text) will cost $0.06 per 1K prompt tokens and $0.12 per 1K completion tokens.
  • The number of parameters and size of the training data set have both not been published. So competitors are not encouraged to replicate these performance ingredients but are referred to a freely available benchmark (OpenAI Evals) that measures the real performance.
  • GPT-4 ranks in the 10% best of the bar exam and 0.5% best of biology olympiad.
  • GPT-4 can handle contexts of over 25,000 words.
  • GPT-4 can access images as inputs and can generate captions, classifications, and analyses. However, this image-to-text functionality is not yet publicly available.
  • Microsoft Bing was already using an early version of GPT-4 in the last few weeks.

An excellent overview by Greg Brockman, President and co-founder of OpenAI, can be found on youtube.

Microsoft released Visual ChatGPT on March 08, 2023, in a paper and with source code on GitHub and Hugging Face. Although this does not seem to be GPT-4-based, it demonstrates similar image capabilities via a combination of pre-existing technologies (generate/modify [text-to-image], and describe [image-to-text]).

Two days after the GPT-4 release, Microsoft announced on March 16, 2023, the integration of GPT-4 into their Office products as a feature they called Copilot. Copilot is not yet available for general use, but Microsoft plans to roll it out gradually to selected customers in the coming months.

OpenAI releases ChatGPT and Whisper APIs

March 2, 2023 / / 0 Comments

On March 01, 2023, OpenAI announced the releases of APIs for ChatGPT (published on Nov 30, 2022) and the automatic speech recognition (ASR) engine Whisper for speech-to-text (STT) transcription (and translation) that was open-sourced in Sept 2022.

The ChatGPT model family is called gpt-3.5-turbo and costs just $0.002 per 1k tokens, which is 10 times cheaper than the existing GPT-3.5 models. Instead of consuming unstructured text as traditionally done by GPT, the ChatGPT models consume a sequence of messages with metadata following a new format called Chat Markup Language (ChatML). The number of tokens (tokens in prompt + tokens in response as available via response[‘usage’][‘total_tokens’]) is restricted to 4096. Notice that there is no possibility to fine-tune gpt-3.5-turbo models.

For Whisper the large-v2 model is now available through an API for a price of $0.006 per minute. The API contains endpoints for transcriptions (transcribes in source language) and translations (transcribes into English).

In addition, the possibility of dedicated instances for professional users was announced that can make economical sense beyond ~450M tokens per day.

A significant change that was made in the Terms of Service and Usage Polices is that data submitted to the API is no longer used for service improvements (e.g. model training) unless an organization opts in. Before it was necessary to opt-out.

OpenAI announced ChatGPT Plus

February 2, 2023 / / 0 Comments

OpenAI announced on Feb 01, 2023, a new subscription plan, ChatGPT Plus, that will be available for $20/month. Benefits are

  • General access to ChatGPT, even during peak times
  • Faster response times
  • Priority access to new features and improvements

ChatGPT Plus will be available first just to customers in the United States.

Since Feb 10, 2023, it is also available in Germany.

On Feb 06, 2023, Google’s CEO Sundar Pichai announced with Bard a competitor to ChatGPT in a message to Google employees. Bard is based on LaMDA (Language Model for Dialog Applications) whose first version was unveiled on May 18, 2021.

Only 2 days later, on Feb 08, 2023, Microsoft’s CEO Satya Nadella demonstrated in a presentation the integration of ChatGPT into Microsoft Bing (codename Sydney).

Microsoft’s VALL-E can synthesize your voice from 3 sec of audio

January 28, 2023 / / 0 Comments

Microsoft has introduced a new language modeling approach for text-to-speech synthesis (TTS) called VALL-E. The approach uses discrete codes derived from an off-the-shelf neural audio codec model, and is trained using 60K hours of English speech, which is hundreds of times larger than existing systems, and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt (project page, paper).

An unofficial Pytorch implementation for VALL-E is available on GitHub.

Google’s Med-PaLM comes close to human performance in clinical knowledge

January 12, 2023 / / 0 Comments

In a recent paper from Dec 26, 2022, Google demonstrates that its large language model Med-PaLM, based on 540 billion parameters with a special instruction prompt tuning for the medical domain, reaches almost clinician’s performance on new medical benchmarks MultiMedQA (benchmark combining six existing open question answering datasets spanning professional medical exams, research, and consumer queries) and HealthSearchQA (a new free-response dataset of medical questions searched online). The evaluation of the answers considering factuality, precision, possible harm, and bias was done by human experts.

GPT-3.5 passes parts of the US legal Bar Exam

January 2, 2023 / / 0 Comments

In the United States, most jurisdictions require applicants to pass the Bar Exam in order to practice law. This exam typically requires several years of education and preparation (seven years of post-secondary education, including three years at an accredited law school).

In a publication from Dec 29, 2022, the authors evaluated the performance of GPT-3.5 on the multiple choice part of the exam. While GPT is not yet passing that part of the exam, it significantly exceeded the baseline random chance rate
of 25% and reached the average human passing rate for the categories Evidence and Torts.
On average, GPT is performing about 17% worse than human test-takers across all categories.

Similar to this publication is the report that ChatGPT was able to pass the Wharton Master of Business Applications (MBA) exam.

On March 15, 2023, a paper was published that stated that GPT-4  significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over GPT-3.5 and beating humans in five of seven subject areas.

RIFFUSION: Stable Diffusion for Real-Time Music Generation

December 16, 2022 / / 0 Comments

By using the stable diffusion model v1.5 without any modifications, just fine-tuned on images of spectrograms paired with text, the software RIFFUSION (RIFF + diffusion) generates incredibly interesting music from text input. By interpolating in latent space it is possible to transition from one text prompt to the next. You can try out the model here.

The authors provide source code on GitHub for an interactive web app and an inference server. A model checkpoint is available on Hugging Face.

There is a nice video about RIFFUSION by Alan Thompson on youtube.

Even more shocking than using diffusion on spectrograms and getting great results may be a paper by Google Research published on Dec 15, 2022. They use text as an image and train their model with contrastive loss alone, thus calling their model CLIP-Pixels Only (CLIPPO). It’s a joint model for processing images and text with a single ViT (Vison Transformer) approach and astonishing performance.

« Older posts Newer posts »

"I was born not knowing and have had only a little time to change that here and there."

Richard P. Feynman

"I have no special talent. I am only passionately curious."

Albert Einstein

"If you tell me precisely what it is a machine cannot do, then I can always make a machine which will do just that."

John von Neumann

© 2024 Stephan Seeger

Theme by Anders Noren — Up ↑