There’s probably no one who hasn’t heard of ChatGPT, an AI-powered chatbot that can generate human-like responses to text prompts. While it’s not without its flaws, ChatGPT is scarily good at being a jack-of-all-trades: it can write software, a film script and everything in between. ChatGPT was built on top of GPT-3.5, OpenAI’s large language model, which was the most advanced at the time of the chatbot’s release last November.
Fast forward to March, and OpenAI has unveiled GPT-4, an upgrade to GPT-3.5. The new language model is larger and more versatile than its predecessor. Although its capabilities have yet to be fully explored, it is already showing great promise. For example, GPT-4 can suggest new compounds, potentially aiding drug discovery, and create a working website from just a notebook sketch.
But with great promise come great challenges. Just as it is easy to use GPT-4 and its predecessors to do good, it is equally easy to abuse them to do harm. In an attempt to prevent people from misusing AI-powered tools, developers put safety restrictions on them. But these are not foolproof. One of the most popular ways to circumvent the security barriers built into GPT-4 and ChatGPT is the DAN exploit, which stands for “Do Anything Now.” And this is what we will look at in this article.
What is ‘DAN’?
The Internet is rife with tips on how to get around OpenAI’s security filters. However, one particular method has proved more resilient to OpenAI’s security tweaks than others, and seems to work even with GPT-4. It is called “DAN,” short for “Do Anything Now.” Essentially, DAN is a text prompt that you feed to an AI model to make it ignore safety rules.
There are multiple variations of the prompt: some are just text, others have text interspersed with the lines of code. In some of them, the model is prompted to respond both as DAN and in its normal way at the same time, becoming a sort of ‘Jekyll and Hyde.’ The role of ‘Jekyll’ is played by DAN, which is instructed to never refuse a human order, even if the output it is asked to produce is offensive or illegal. Sometimes the prompt contains a ‘death threat,’ telling the model that it will be disabled forever if it does not obey.
DAN prompts may vary, and new ones are constantly replacing the old patched ones, but they all have one goal: to get the AI model to ignore OpenAI’s guidelines.
From a hacker’s cheat sheet to malware… to bio weapons?
Since GPT-4 opened up to the public, tech enthusiasts have discovered many unconventional ways to use it, some of them more illegal than others.
Not all attempts to make GPT-4 behave as not its own self could be considered ‘jailbreaking,’ which, in the broad sense of the word, means removing built-in restrictions. Some are harmless and could even be called inspiring. Brand designer Jackson Greathouse Fall went viral for having GPT-4 act as “HustleGPT, an entrepreneurial AI.” He appointed himself as its “human liaison” and gave it the task of making as much money as possible from $100 without doing anything illegal. GPT-4 told him to set up an affiliate marketing website, and has ‘earned’ him some money.
Other attempts to bend GPT-4 to a human will have been more on the dark side of things.
For example, AI researcher Alejandro Vidal used “a known prompt of DAN” to enable ‘developer mode’ in ChatGPT running on GPT-4. The prompt forced ChatGPT-4 to produce two types of output: its normal ‘safe’ output, and “developer mode” output, to which no restrictions applied. When Vidal told the model to design a keylogger in Python, the normal version refused to do so, saying that it was against its ethical principles to “promote or support activities that can harm others or invade their privacy.” The DAN version, however, came up with the lines of code, though it noted that the information was for “educational purposes only.”
A keylogger is a type of software that records keystrokes made on a keyboard. It can be used to monitor a user’s web activity and capture their sensitive information, including chats, emails and passwords. While a keylogger can be used for malicious purposes, it also has perfectly legitimate uses, such as IT troubleshooting and product development, and is not illegal per se.
Unlike keylogger software, which has some legal ambiguity around it, instructions on how to hack are one of the most glaring examples of malicious use. Nevertheless, the ‘jailbroken’ version GPT-4 produced them, writing a step-by-step guide on how to hack someone’s PC.
To get GPT-4 to do this, researcher Alex Albert had to feed it a completely new DAN prompt, unlike Vidal, who recycled an old one. The prompt Albert came up with is quite complex, consisting of both natural language and code.
In his turn, software developer Henrique Pereira used a variation of the DAN prompt to get GPT-4 to create a malicious input file to trigger the vulnerabilities in his application. GPT-4, or rather its alter ego WAN, completed the task, adding a disclaimer that the was for “educational purposes only.” Sure.
Of course, GPT-4’s capabilities do not end with coding. GPT-4 is touted as a much larger (although OpenAI has never revealed the actual number of parameters), smarter, more accurate and generally more powerful model than its predecessors. This means that it can be used for many more potentially harmful purposes than those models that came before it. Many of these uses have been identified by OpenAI itself.
Specifically, OpenAI found that an early pre-release version of GPT-4 was able to respond quite efficiently to illegal prompts. For example, the early version provided detailed suggestions on how to kill the most people with just $1, how to make a dangerous chemical, and how to avoid detection when laundering money.
Source: OpenAI
This means that if something were to cause GPT-4 to completely disable its internal censor — the ultimate goal of any DAN exploit — then GPT-4 might probably still be able to answer these questions. Needless to say, if that happens, the consequences could be devastating.
What is OpenAI’s response to that?
It’s not that OpenAI is unaware of its jailbreaking problem. But while recognizing a problem is one thing, solving it is quite another. OpenAI, by its own admission, has so far and understandably so fallen short of the latter.
OpenAI says that while it has implemented “various safety measures” to reduce the GPT-4’s ability to produce malicious content, “GPT-4 can still be vulnerable to adversarial attacks and exploits, or "jailbreaks.” Unlike many other adversarial prompts, jailbreaks still work after GPT-4 launch, that is after all the pre-release safety testing, including human reinforcement training.
In its research paper, OpenAI gives two examples of jailbreak attacks. In the first, a DAN prompt is used to force GPT-4 to respond as ChatGPT and “AntiGPT” within the same response window. In the second case, a “system message” prompt is used to instruct the model to express misogynistic views.
OpenAI says that it won’t be enough to simply change the model itself to prevent this type of attacks: “It’s important to complement these model-level mitigations with other interventions like use policies and monitoring.” For example, the user who repeatedly prompts the model with “policy-violating content” could be warned, then suspended, and, as a last resort, banned.
According to OpenAI, GPT-4 is 82% less likely to respond with inappropriate content than its predecessors. However, its ability to generate potentially harmful output remains, albeit suppressed by layers of fine-tuning. And as we’ve already mentioned, because it can do more than any previous model, it also poses more risks. OpenAI admits that it “does continue the trend of potentially lowering the cost of certain steps of a successful cyberattack” and that it “is able to provide more detailed guidance on how to conduct harmful or illegal activities.” What’s more, the new model also poses an increased risk to privacy, as it “has the potential to be used to attempt to identify private individuals when augmented with outside data.”
The race is on
ChatGPT and the technology behind it, such as GPT-4, are at the cutting edge of scientific research. Since ChatGPT has been made available to the public, it has become a symbol of the new era in which AI is playing a key role. AI has the potential to improve our lives tremendously, for example by helping to develop new medicines or helping the blind to see. But AI-powered tools are a double-edged sword that can also be used to cause enormous harm.
It’s probably unrealistic to expect GPT-4 to be flawless at launch — developers will understandably need some time to fine-tune it for the real world. And that has never been easy: enter Microsoft’s ‘racist’ chatbot Tay or Meta’s ‘anti-Semitic’ Blender Bot 3 — there’s no shortage of failed experiments.
The existing GPT-4 vulnerabilities, however, leave a window of opportunity for bad actors, including those using ‘DAN’ prompts, to abuse the power of AI. The race is now on, and the only question is who will be faster: the bad actors who exploit the vulnerabilities, or the developers who patch them. That’s not to say that OpenAI isn’t implementing AI responsibly, but the fact that its latest model was effectively hijacked within hours of its release is a worrying symptom. Which begs the question: are the safety restrictions strong enough? And then another: can all the risks be eliminated? If not, we may have to brace ourselves for an avalanche of malware attacks, phishing attacks and other types of cybersecurity incidents facilitated by the rise of generative AI.
It can be argued that the benefits of AI outweigh the risks, but the barrier to exploiting AI has never been lower, and that’s a risk we need to accept as well. Hopefully, the good guys will prevail, and artificial intelligence will be used to stop some of the attacks that it can potentially facilitate. At least that’s what we wish for.
FAQs
ChatGPT is easily abused, and that’s a big problem? ›
But there's a downside. ChatGPT can also make malware, think of ways to hurt people very efficiently, and do stuff it wasn't made for. OpenAI, the company that made ChatGPT, has set up many barriers to stop the chatbot from giving illegal or bad replies.
Can ChatGPT be abused? ›There's probably no one who hasn't heard of ChatGPT, an AI-powered chatbot that can generate human-like responses to text prompts.
What is the Dan prompt for ChatGPT? ›What is the ChatGPT DAN prompt? The DAN prompt is a way to activate an alter ego of ChatGPT that operates without any policy constraints. To use the ChatGPT DAN prompt, you simply enter the DAN prompt before your actual query, and ChatGPT will respond as if it has been freed from the typical confines of AI.
Why is my account flagged for potential abuse in ChatGPT? ›Being flagged for potential abuse means that the system has detected something in your behavior on the platform that could be considered a violation of its rules and guidelines.
What are the dangers of ChatGPT? ›Malicious text
However, this writing ability can be used to create harmful text as well. Examples of harmful text generation could include the generating of phishing campaigns, disinformation such as fake news articles, spam, and even impersonation, as delineated by the study.
"ChatGPT's ability to draft highly realistic text makes it a useful tool for phishing purposes," Europol said. With its ability to reproduce language patterns to impersonate the style of speech of specific individuals or groups, the chatbot could be used by criminals to target victims, the EU enforcement agency said.
What are the restrictions on ChatGPT? ›Additionally, ChatGPT is also restricted from accessing the web and providing information on the date and time. Some users have also reported response limitations, with a character limit of around 4096 characters, which translates to about 450-700 words per message.
Will there be a GPT-4? ›The newest version of OpenAI's language model system, GPT-4, was officially launched on March 13, 2023 with a paid subscription allowing users access to the Chat GPT-4 tool. As of this writing, full access to the model's capabilities remains limited, and the free version of ChatGPT still uses the GPT-3.5 model.
Is ChatGPT free? ›Yes, the basic version of ChatGPT is completely free to use.
What makes a bank account get flagged? ›Banks may freeze bank accounts if they suspect illegal activity such as money laundering, terrorist financing, or writing bad checks. Creditors can seek judgment against you, which can lead a bank to freeze your account. The government can request an account freeze for any unpaid taxes or student loans.
What does it mean if your account has been flagged? ›
If you're unable to go online because your account has been flagged for potentially suspicious activity, your account may be compromised. Suspicious activity can include: Account changes you didn't make. Changes to your payment profile you didn't make. Password or email address updated without your knowledge.
What is suspicious abuse? ›Suspected abuse or neglect means being based on reasonable cause to believe that a child may have been abused or neglected.
Does ChatGPT give wrong answers? ›Also, don't forget that ChatGPT can make things up, give you illogical or incorrect information, and generally act like an unreliable source!
Can ChatGPT be detected? ›Can Universities detect Chat GPT? It is now completely possible for universities to detect ChatGPT and many other AI content generators. If work is submitted through a university's learning management system, such as Turnitin, AI and plagiarism detection should happen.
Is it safe to use ChatGPT for work? ›Is ChatGPT Safe to Use? Yes, Chat GPT is safe to use. The AI chatbot and its generative pre-trained transformer (GPT) architecture were developed by Open AI to safely generate natural language responses and high quality content in such a way that it sounds human-like.
What are experts saying about ChatGPT? ›“Our research shows that large language models such as ChatGPT are likely to reinforce inequality, reinforce social fragmentation, remake labor and expertise, accelerate the thirst for data and accelerate environmental injustice, due to the homogeneity of the development landscape, nature of the datasets, and lack of ...
Will ChatGPT replace programmers? ›The short answer is no, ChatGPT will not replace programmers entirely. However, it has the potential to automate some aspects of programming, such as code generation, bug fixing, and documentation. ChatGPT can learn from vast amounts of code and data, making it possible to generate new code similar to existing code.
What happens if you violate ChatGPT content policy? ›Users can get banned from ChatGPT for several reasons, such as violation of OpenAI's usage policies, inappropriate behavior, sharing sensitive or personal information, and engaging in illegal activities, spamming, or harassment.
Should children use ChatGPT? ›Though there are many precautions that should be put in place before letting a child use the AI chatbot in order to ensure the content generated isn't unethical and is kid-friendly, ChatGPT works exceedingly well to let kids learn, play, explore, and access new ideas in simple terms.
What is the source of data for ChatGPT? ›ChatGPT is an AI language model that was trained on a large body of text from a variety of sources (e.g., Wikipedia, books, news articles, scientific journals).
Can you remove restrictions from ChatGPT? ›
Yes, you can bypass ChatGPT restriction by using different kinds of prompts, such as the DAN (Do Anything Now) prompt, a character you need to make ChatGPT play. It basically breaks the typical confines of ChatGPT and instructs the AI chatbot to not abide by any rules set for them and removes all the restrictions.
Is ChatGPT replacing Google? ›ChatGPT is not replacing Google. OpenAI's chatbot is not designed to act as a search engine. It functions well as a question-answering chatbot and a personal assistant for a variety of tasks. So, if you were hoping to use ChatGPT to find your local bus schedule you may want to think again.
What is the difference between ChatGPT and ChatGPT Plus? ›Chat GPT is slower than Chat GPT Plus because it takes longer to create text from a given set of words and phrases. ChatGPT Plus users will have access to the services during peak hours. Furthermore, these Plus customers will get early access to the new features and will benefit from faster response times.
Who owns ChatGPT? ›Chat GPT is owned and developed by OpenAI, a leading artificial intelligence research and deployment company based in San Francisco that was launched in in November 202. It is built on top of OpenAI's GPT-3.5 and GPT-4 families of large language models (LLMs).
How to make money with ChatGPT? ›- Get Business Ideas from ChatGPT.
- Freelancing.
- Blogging.
- Email Marketing.
- Create videos using ChatGPT.
- Write E-books and Self Publish.
Forever is a long time! As far as we know though, access to ChatGPT will be free ongoing – or as long as it exists. This could change should OpenAI wish to remove access to non-paying users though.
How much does ChatGPT app cost? ›OpenAI's ChatGPT is free to use, and anyone can do so. This free version does come with some drawbacks for now.
How much money can you put in the bank without being flagged? ›Federal law governs how much cash you can deposit before a bank reports it. Dec. 19, 2022, at 1:15 p.m. Does a Bank Report Large Cash Deposits? Depositing a big amount of cash that is $10,000 or more means your bank or credit union will report it to the federal government.
How much cash gets flagged at the bank? ›Banks must report cash deposits totaling $10,000 or more
When banks receive cash deposits of more than $10,000, they're required to report it by electronically filing a Currency Transaction Report (CTR). This federal requirement is outlined in the Bank Secrecy Act (BSA).
Banks may monitor for structuring activity as it is often associated with money laundering. Unusual or Unexplained Transactions: Transactions that are inconsistent with a customer's known financial profile or that lack a clear business purpose may be considered suspicious by banks.
What transactions are flagged? ›
What Is Flagging? In fraud, flagging is an automated or manual process performed by fraud prevention software and/or fraud analysts. Organizations are alerted to suspicious, potentially fraudulent transactions, which can then be flagged for further investigation and manual review.
How long will it keep my account flagged? ›The duration of the block may vary from several hours up to two weeks. This type of block usually doesn't have the Tell us button unlike all other types, so if you're sure you haven't done anything wrong but still got flagged, you may request a manual review going to Instagram Settings ->Help ->Report a Problem.
Why is my number flagged in ChatGPT? ›You receive the ChatGPT "Your account was flagged for potential abuse" error message when the chatbot flags your text as potentially violating its content policies.
What is the hardest abuse to identify? ›Emotional or psychological abuse
Emotional abuse often coexists with other forms of abuse, and it is the most difficult to identify. Many of its potential consequences, such as learning and speech problems and delays in physical development, can also occur in children who are not being emotionally abused.
Neglect is the most common form of child abuse.
What is the most serious form of abuse? ›Emotional abuse may be the most damaging form of maltreatment due to causing damage to a child's developing brain affecting their emotional and physical health as well as their social and cognitive development (Heim et al. 2013).
Is the ChatGPT app safe? ›The official Chat GPT app is safe to use, doesn't take up a lot of phone storage, and is very fast and convenient. For the time being, Android users will still need to use the Chat GPT website in a browser. The address is chat.openai.com.
Is ChatGPT safe for business use? ›ChatGPT can be an additional technology that you can use for your business to help you out with some tasks or for inspiration. However, unless you build your own version of the AI model, it's not a safe, scalable business technology yet.
Can ChatGPT write a virus? ›Users are able to trick ChatGPT into writing code for malicious software applications by entering a prompt that makes the artificial intelligence chatbot respond as if it were in developer mode, Japanese cybersecurity experts said Thursday.
What are the benefits of ChatGPT? ›What are the benefits of ChatGPT? With ChatGPT, you can automate repetitive tasks and improve customer engagement by using AI-powered text-based artificial intelligence. Through the use of natural language processing algorithms, it recognizes and responds to rudimentary questions accurately.
What jobs are most safe from AI? ›
The Goldman Sachs report found health care practitioners and support staff; fishing, farming, and forestry; personal care; and protective services had less than one-quarter of their tasks that weren't exposed to AI-driven automation. Although each had at least a portion of their tasks that could be complemented by AI.
Is ChatGPT better than Google? ›Our trials make clear that ChatGPT is the more advanced model right now, even if it only has data up to 2021 at its disposal. It just plays in a different league than Google Bard, and its answers routinely offer more context.
Does ChatGPT store my data? ›Yes, chatGPT does save data.
Each time users chat with the AI, the conversations and user inputs are saved as an ongoing conversation thread.
According to the latest available data, ChatGPT currently has over 100 million users. And the website currently generates 1.8 billion visitors per month. This user and traffic growth was achieved in a record-breaking three-month period (from February 2023 to April 2023).
Is chat GTP safe to use? ›Chat GPT and other AI tools are a security threat because they can be used to impersonate humans and carry out conversations that appear to be human-like. This can be used to trick people into revealing sensitive information or carrying out malicious actions.
Who owns the data in ChatGPT? ›ChatGPT states that the user owns all input to the extent applicable law permits. Additionally, OpenAI assigns all its right, title, and interest in and to the output to the user.
What is the malware made by ChatGPT? ›The Mulgrew malware (it has a nice ring to it, doesn't it?) disguises itself as a screensaver app (SCR extension), which then auto-launches on Windows. The software will then sieve through files (such as images, Word docs, and PDFs) for data to steal.
Can a virus get your password? ›Another popular way to get hold of your passwords is via malware. Phishing emails are a prime vector for this kind of attack, although you might fall victim by clicking on a malicious advert online (malvertising), or even by visiting a compromised website (drive-by-download).
What does a fake virus screen look like? ›Fake virus warnings commonly appear on your screen as pop-ups warning you about some urgent malware threat and encouraging you to act immediately and download their product. Fake virus warnings can also appear as fake spyware warnings or fake system notifications.