How LLM Function Calling Works: Principles and Implementation

I Was So Furious—Someone Finally Explained This Clearly

Do you have any idea how broken I was back in June 2023?

A user casually throws out, "What's the weather like in Beijing today?" and I’ve got to write a regex, like I’m performing surgery, to surgically extract the word "Beijing" from a clump of Chinese characters. Then call the weather API, plug the data back into the model, and pray it doesn’t screw up.

And guess what?

I explicitly wrote "Please output JSON" in the prompt—and it insisted on replying, "Sure, I'll check for you! ☀️"

That little smiley face sent me over the edge.

Clunky, dangerous, and crashing all the time.

Back then, I saw a rant on Zhihu that I still remember word for word: "I wrote my function description longer than my resume, and the model still called the wrong thing."

Heartbreaking, right?

Back Then, How Did We Even Survive? Pure Luck

The earliest solution was basically "tuning by superstition."

You’d repeat in the system prompt: "If the user asks for real-time info, output a JSON where the key is 'function' and the value is the parameter list."

And then?

You'd clasp your hands together and beg the heavens: please don’t wrap the JSON in a Markdown code block, don’t add extra quotes to the parameters, don’t suddenly blurt out, "I’m not really sure, but I think I should call this function…"

Speaking of which, there’s one incident that still sends chills down my spine.

I was testing a "query order" tool, and the user just casually asked, "Is your company reliable?"

And guess what the model did?

It directly output my internal query_order(status="cancelled")!

Privacy? Gone!

That so-called "tool calling" back then was nothing but a coin toss. When luck was on your side, the output format was correct. When luck wasn't, it just broke completely.

June 2023: The Bomb Dropped

Right around that time, OpenAI quietly added a functions parameter to the GPT-4 API.

Honestly, my first reaction was—"Yeah, right, another pipe dream."

But I still gave it a try, just out of curiosity.

And then I was completely floored.

I registered a get_weather function with two parameters, location and unit, and asked: "What’s the temperature in Boston right now?"

It didn’t return some rambling text. It returned a clean, structured JSON:

{
 "name": "get_weather",
 "arguments": "{\"location\": \"Boston\", \"unit\": \"fahrenheit\"}"
}

See, it didn’t go fetch the weather itself. It just told me: "I want to call this function with these parameters."

I took that JSON, parsed it, called the API, got the result, fed it back to the model, and only then did the model produce a natural language answer.

The whole process can be summed up in one sentence: The model only makes the decision; it doesn't carry out the execution.

And with that, all the chaos vanished.

Because now the format was enforced by the API, not coaxed out of the prompt. The model had been specifically fine-tuned to know when to output a call instruction and when not to.

A colleague once asked me: "So how does it know whether to call or not?"

That’s the million‑dollar question.

If you’ve read OpenAI’s official blog, you’ll know that the Function Calling model went through SFT and RLHF training—including reinforcement learning from AI feedback, where one AI acts as a judge to score candidate responses, instead of relying entirely on humans.

Low cost, fast, and scalable for batch training.

Through tons of feedback, the model learned its boundaries: if the user asks "What’s your name?" just answer directly, no tool needed. If the user asks "Check some real-time data for me," then jump in.

That sense of boundaries? SFT alone can’t teach it. It has to be hammered in through repeated feedback signals.

The Bloody Traps I Fell Into

My first mistake when I started writing Function Calling was making function descriptions too abstract.

For example, I wrote the description for get_weather as: "Get weather info."

Result? The user asks, "Is it good for running tomorrow?" and the model stubbornly refuses to call the weather tool—because "good for running" and "get weather info" have nothing semantic in common.

Later I changed the description to: "Get current weather or forecast for a given location and date. Used to answer weather-related questions (e.g., is it raining, temperature, suitable for outdoor activities)."

Suddenly, the model got it.

And then there’s the parameter description pit. I had a parameter called time and just wrote "time." The model took the user’s "the day after tomorrow" and literally passed the string "the day after tomorrow" without converting it to a proper date format.

So I added a detailed description: "Date in YYYY-MM-DD format. Supports relative dates like 'the day after tomorrow' which will be automatically converted." I also set the format to "date."

Then the model learned to output a proper format.

Another trap that made my blood boil: the model would hallucinate results that didn’t exist.

I was testing a search_web tool, and before I even executed the function, the model started answering the question.

What was the problem? I hadn’t set function_call="auto"—I left it empty. No wonder it went off the rails.

After I set function_call="auto", the model learned to output a call instruction when needed and answer normally otherwise. This feature only became truly stable after GPT-3.5-turbo-1106.

How Did Others Follow Up?

Function Calling isn't exclusive to OpenAI.

Anthropic added it in Claude 2.1, almost the same format, except they call the parameter tools instead of functions. Google’s Gemini series supports it too, with an extra tool_call field in the message structure.

On the open‑source side, Llama 3.1 and Qwen 2.5 implemented similar capabilities through fine‑tuning.

I actually took the time to compare the source code differences between ChatGLM2 and ChatGLM3. The changes were actually small—add a few special tokens (like <|assistant|>), plus some logic to handle function schemas.

The usage is almost a mirror image of OpenAI’s, with the same schema structure.

But what’s interesting is that different models have wildly different "calling tendencies."

Some models will call the calculator tool even when you ask "What’s 2+2?"—clearly overtrained toward calling. Others are the opposite: they stubbornly rely on their training knowledge when they should be calling a tool, and then hand you a hallucination.

Behind this is still the problem of training data balance—something you can’t fix by tweaking a few lines of description.

One Call Isn’t Enough? Use a Loop

A single Function Calling can’t handle complex scenarios.

For example, the user says: "Check tomorrow’s weather in Shanghai. If it’s over 30°C, send me an email reminder."

How do you do that? First call the weather tool, then based on the result decide whether to call the email tool. A complete Agent flow means putting Function Calling inside a loop—the model outputs a call instruction, the code executes, the result is fed back, the model judges again, possibly calls again, until it finally answers the user.

I fell into a trap here too.

One time, the model saw the weather was over 30°C, called the email tool, and the email was sent successfully. And guess what? Then it called the email tool again, saying, "I want to double‑check."

The user got two identical emails.

The fix was simple: add a line in the system prompt like "Each tool can only be called once, do not call repeatedly," and implement idempotency checks in code—record which functions have been called and their results, so that a repeated call returns the cached result directly.

That’s the basic implementation of an Agent: not a one‑shot decision, but a loop of decision, execution, and feedback.

Function Calling is the building block of that loop.

Now and the Future

Today, almost all my production code involving large language models uses Function Calling.

The industry is