Primary Technology

welcome to a special episode of Primary
Technology.

I'm here with indie developer Ryan Wesley,

the founder maker of Food Noms,

which is an amazing food tracking app.

The first one I've stuck with long term,

been using it for over a year,

and it is incredible and had a huge update
for I was twenty six.

And now I want to ask him about Siri AI
and app.

So, Ryan, thanks so much for joining me
today.

Yeah, thanks, Stephen.

So I posted on Mastodon that as I was
making a video about third-party apps

and Siri AI, you had a lot of thoughts and
explanation about what's happening there.

And even as I started making my video,

I was like, Man, all these apps say they
work with Siri AI and they've been
updated,

but trying to get Siri AI to use them can
is a very mixed bag.

And a lot of times I I don't know what's
happening.

Sometimes it'll ask, Do you want to enable
these shortcuts actions?

But it's not really a shortcut.

So

If we could start from the top,

maybe you can explain when an app
developer right now is wanting to

say their app works with Siri AI,

what does that mean from an implementation
standpoint?

Like what are you doing to an app so that
it works with Siri AI today?

for most developers what they would say is
that they are using

App Intense there's kind of App Intense is
like this big umbrella thing

and there's a lot of things that fall
underneath it and some of the newer stuff

is what they call like schema domains.

And so Apple has like published these new
APIs that are

Tailored for certain like verticals,

certain like specific kinds of apps.

So like calendar apps, to-do apps,

reminder apps, photo editing apps.

there's kind of like I don't know how
many,

like a dozen different categories.

And those are if if your app like fits one
of those verticals,

then you can basically just like put in
the code that like

you know, this is my data model.

This is what like I'm a journaling app,

right? And so this is a journal is is
corresponds to this data model I have.

And like if you want to create the
journal,

it's like basically like a template,

and then like the developer all they have
to do is kind of glue in the

the code for actually like executing

That template. I

from

the user perspective, like let's say a
developer implements those that those data

points, that schema for one of those
categories.

The

Yeah.

ideal is that then the user can ask the
voice assistant,

the new one, Siri AI, what did what what
was my calorie goal yesterday?

And like in a perfect world, it would know
that I use food noms and it should look

Yeah.

at that schema and pull the number and
just tell me.

but it's

maybe not that simple in actual the
implementation.

Right? Is that right?

Right.

I mean I mean part of the part of the
issue is that like there is

no food tracking schema. So in order for
me to do anything,

I have to say like, I'm actually like a
journal app or like a calendar app,

something that I'm not really.

And then like I have to like basically
shoehorn my actions into like whatever

actions are available. And there's not
really many.

It's like basically crud, create,

read, update, delete. Those are like

what you have what you have at your
disposal.

And even beyond that, like say there's
something specific like,

~ well, for food noms, it's like there's a
meal type,

right? Breakfast, lunch, dinner.

That's like a pi additional metadata that
goes on to like if let's say

I use the journal entry, like that's not
Apple doesn't have meal types for
journals.

So like that sort of metadata just gets
left out.

So for me, there's not really

a great option, unfortunately.

And I think Apple's like answer to that
today is like,

well, we're going to be expanding and
adding more schema domains in the future.

And so I just have to ha kind of hope for
that.

There is one other kind of solution that
they have told me in the past

is like the way to go. If you have an app
that doesn't work with a specific domain,

there is something called app shortcuts.

curious, do you know what app shortcuts
are?

Well, so my mind goes to like the
shortcuts app,

but I assume that's not it. And I think
that's what like in my video when

I was trying to get information from the
Flighty app,

I got this

Mm-hmm.

weird prompt from Siri AI that said,

Do you want to enable shortcuts?

And even as a pretty I would say power
user of shortcuts,

I didn't really know what that meant.

And I assumed like, is this describe a
shortcut?

Is this about to vibe code a shortcut for
me?

But it seemed like that is not the case.

Instead it was enabling some mechanism and
then immediately showed

me flight information. So I guess I'm
wrong that it is not like the shortcuts

we all know and love. What what is that?

Well, it is sort of related.

Okay.

app shortcuts I think actually are really
cool.

And you've probably used them before.

I mean, I know you know the shortcuts app
pretty well.

If you if you go like to the shortcuts app
and you kind of like you scroll down

to the app section, right, and you

Right, right.

just select

an app, there's like those defaults that
just come like kind of pre-installed.

Right. And you can just like one tap.

You don't have to make a custom shortcut
for them to work.

They just work kind of out of the box,

right? those are the app shortcuts.

So that's

Okay.

like another API. And they've actually
existed for quite a while.

And they're actually I I love that the
concept.

It's ~ actually in my mind like kind of a
a very elegant design.

And it's pretty simple to set up too as a
developer.

So you have like your

your app intents, like your shortcuts,

you know, if you support shortcuts as a
developer,

you have like all those actions that you
can offer to your users.

An app intent, I mean an app shortcut
basically wraps an app intent and kind

Okay.

of like simplifies it. And then you add in
like some some phrases and those phrases,

you just list out like a series of sort of
like things someone could say

to Siri and it would invoke that app
intent.

So for food noms that that opens up I have
like I think ten or

so different app shortcuts. There is a
limit on how many you can offer,

but you know, there's things like check my
food noms protein goal or like

log water in food noms because if you have
water as like a favorite,

then you can do that. Or you can say copy
my lunch from yesterday to today

or open the food nom scanner or like
there's these are all these like really
rich

capabilities.

That ~ and I do think that some of the
developers who are claiming series

AI support, they are kind of talking about
app shortcuts.

because they do work, but not reliably.

And I think I saw that in your video,

that like

Yes.

it it kind of depends on a series of
factors,

and it from my investigation it comes down
to just like a handful of

or more of like just honestly bugs in the
operating system that's causing these

to not work reliably.

So that brings me to my next question.

So I think I understand that. And for like
Flighty,

when I said yes, use the app shortcuts.

Yeah.

Now it can show me my flight. But there
were a number of instances where

I would ask it something that I know
sometimes it'll show me the app result.

And like one is like Lumi, L U ~ Y,

which will do things like tell you about
golden hour,

what time that is, the moon phase.

And if I ask Siri AI, tell me what golden
hour is from Lumi.

Or I've seen other guidance where you say
the words ask Lumi,

like I'm telling Siri AI that and it
should reach into the Lumi app

and tell me that. It's like fifty-fifty.

Sometimes

Mm-hmm.

I'll see the

little Lumi app logo in the Siri AI result
and it will tell me what the Lumi

AI is saying or Lumi app is saying.

And the other half of the time it'll give
me an answer,

but the sources are clearly like web
search or like the general world

Yeah.

broad world knowledge that Siri AI has.

In your estimation, like what is happening
there?

Is it that Siri AI is trying to interpret
the request and even if I

say the app name, it's like doing
something different?

Like what what do you think is happening
there?

From what I can tell, and it is kind of
hard,

I'm having to look at like the logs of the
operating system.

So I connect my phone and like I stream
the in all the logs of the whole OS.

And I'm doing this with Claude is is
helping me because it's just like

so many logs it helps like to filter it
out and kind of make sense of it all.

Honestly, like my kind of general take on
it is that Siri AI and like when

you interact with Siri, like you're still
interfacing with like

There are so many layers in the OS.

And these like some of these layers I
think are really,

really old. Like it's not like they they
like completely started fresh with Siri,

with Siri AI. There are still so many like
frameworks and like demons

and things going on that are like pr
trying to process that request.

And it's just like there's so many failure
points along that way.

So depending on like yeah, like

I've noticed differences from if you like
type it versus if you say it,

because like there's like that's like one,

you know, like part of the the pipeline of
processing the requests.

It's just like there's different failure
points and in my opinions,

like bugs along the way to processing
those requests,

and it's just yeah, unfortunately
unreliable.

So sometimes it it doesn't it like I don't
know why.

Like if you explicitly say the name.

I don't understand why it's not like like
that's pretty clear cut.

Like we should handle

Yeah.

this with the the associated app shortcut.

Like it should work that way. For some
reason it doesn't.

There's just too many failure points.

And that's something where I've I've
noticed with notes,

which is clearly something that Siri AI
can use third party apps with.

Jason A10, ~ my co-host on this show,

contextually, a lot of times I can ask it
to do something and contextually it'll

do it. With bare notes, again,

it's like 75% of the time. If I say create
a bare note with this information,

it'll do it. If I ask tell me about John
Turnus,

Apple CEO, it gives me a Siri result with
world knowledge.

And then I say,

add this to a bear note or create a new
bear note with this information,

it'll just create an Apple note.

And like there's

Mm-hmm.

not like I can't

I can't get it to like shove it into bear
instead of notes.

And then sometimes it'll ask me.

And so sometimes I'll well say like add a
new reminder to take out the trash

at 8 p.m. And sometimes it'll just create
a reminder.

Sometimes it will give me a list of 10
apps that I have on my phone,

like tasks, can do things, the stock
reminders app,

and I have to pick one.

And I guess and you tell me if Apple has
instructed developers in this way.

Is Siri AI supposed to learn my habits
over time so that if I always choose

can do for my tasks, that a month from
now,

after I've done dozens of requests,

it'll stop asking me and it'll just
default to that third party app?

Or is that un is that an unknown process?

I don't know. I don't know. Yeah.

Okay. So th so there's not any guidance
from Apple of like tell your users do
this,

You know,

right?

no, I mean I haven't heard anything.

but y you know, I I may be missing
something to be honest.

I I actually haven't like so with the with
the Siri Domain stuff that

I was talking about, and that is like I
think the the apps like Bear

and like Can Do, right? Like that you're
able to take advantage of those,

like

I wasn't able to, so I didn't spend as
much time like investigating

and and like learning about all the
intricacies of that and and so

for that like that example, you know,

that might be a gap in my knowledge.

Or I don't know, I wouldn't be surprised
if it's just an omission

or like Apple hasn't been clear about like
exactly how that should work.

I think generally they try

So he's yeah.

to avoid committing to certain behaviors.

They they

Right. And and

yeah, to the to like to their credit,

I imagine like because Siri AI is doing so
much now,

like it takes a week to index your phone
and it might it's reaching your personal

context and world knowledge and third
party apps,

like it's complex, you know, t d what it's
all trying to do.

Do you does it feel like maybe some apps
like you're saying didn't

fit nicely into one of those twelve
schemas for Siri AI,

like journaling, tasks, notes,

Mm-hmm.

that some are trying to use those app
shortcuts

To kind of say they have Siri AI support,

but that is why some users might get a
mixed experience because it's actually

not using the a proper schema and Siri AI
then can't really parse that this request

is for this kind of app. Is that fair to
say?

A lot. Yeah. Okay.

Yeah, likely. Yeah.

All right.

I I think yeah, there's there's other
issues.

Like I think you've pointed out,

like even when you have a schema,

there's like maybe some bugs that they
need to work out or some limitations that

are causing like potential user confusion
or or bad experiences.

And I think y I did see some of those in
in your video.

But ~ yeah, I think there is general
confusion when like an app says they have
Siri

A AI support and instead it's really app
shortcuts.

That happen to be working well for them.

You know, there's some app shortcuts for
me that just like happen

to work better than others. And I I don't
exactly know why.

It's something about I think the natural
language processing and like

the way the phrase is constructed.

Like it just trips up Siri. And again,

like those like multiple layers of
processing.

and like maybe like Flighty, you know,

the I think you could probably engineer
like and try to figure

out the perfect phrase that has like a
high success rate.

Just like the magic incantation.

And ~ yeah,

Yeah.

I th I guess the risk there is that like a
user then like has

to remember that incantation for it to
work reliably.

And if they start like to like say it
slightly differently or ask

for a different like feature in Siri AI,

they may get something like totally
unexpected,

right? Like a web general knowledge

Yeah.

response.

And I actually use those words exactly in
my video was like,

I guess I gotta find the magic words
because sometimes

Ha ha.

like if I find a phrase, like sometimes
saying ask Flighty about my next trip

or ask Mercury weather about my next trip,

it would work consistently. But if I asked

Mm-hmm.

what's my next trip in Mercury weather,

it'd be like, No, sorry You know,

and it's one of those things

Yeah.

where the promise is supposed to be
natural language and that's always where
Apple

has tried to err on, where it will take
kind of general

broad phrases and still try to get up
where you're going.

But when you replied ~ to my video on
Mastodon,

I mentioned I asked just to see if Food
Knobs was in Siri AI,

because I assumed like, maybe it'll it'll
have a feature.

And of course you did not advertise that
you have Siri AI support,

but you said in our thread, here's what
you would hope to happen.

What do you what has to happen or change
for you to feel comfortable

to say Food Knobs has Siri AI?

Is it those schema that actually

works with food tracking type apps
specifically?

Is it more predictable commands and
controls?

What are you looking for as a developer?

what would be completely ideal would be
something completely open-ended,

like true MCP style support. That would be
best case scenario.

And because like I offer like FoodNums has
an MCP beta right now,

and so like you can connect Claude to your
FoodNums data and you know,

people are able to do really cool stuff
with it,

like stuff I hadn't even thought of.

Even though like it's the same sort of
primitives that I offer with shortcuts,

you know, like being able to like find
your food logs and like process them

and then like but like because quad,

you know, is not restricted to certain
domains,

right? Like that would be ridiculous.

Quad MCP is like open ended. it's able to
like just do whatever the user like dreams

of and like there's no restrictions on how
the user requests it,

like the

LLM knows how to handle the request
properly.

That would be like best case scenario.

And I don't know, maybe someday that that
comes to fruition.

I think in the short term, what I would
love to see is you know,

I have, like I said, I've built out these
app shortcuts,

which I really do think are like genuinely
valuable.

I'd love to be able to just ask Siri
anywhere,

like, how am I doing on my protein goal?

you know, how many how many grams of
protein do I need to eat today to hit my
goal?

you know, like also just being able to
log,

you know, stuff like really easily,

at least my favorites, right? If I if I
set up,

you know, my snacks that I tend to eat or
like my my favorite meals to just

be able to ask Siri to log them,

you know, that's basically already there,

just waiting to be used. And it's,

you know, I built that like a year and a
half ago,

two years ago.

And I've been like waiting for Apple to
like kind of fix these issues.

And I was like just frankly kind of
disappointed that like I was hoping with
this

big Siri AI relaunch that they would like
make that more reliable.

And two, it was such an opportunity,

right? Like 'cause I write out those
phrases about like the incantations.

You know,

For it.

you can imagine an LLM can just like
basically learn from those phrases

and be able to like turn it into like

Any sort of like related query gets
automatically funneled to that

app shortcut and it would just work.

And I really think that app shortcuts are
just a great solution for them

in terms of like the long tail too.

Like they're never going to be able to
provide like domains for every single kind

of app. So like but every app can create
custom intents and then

app shortcuts on top of it. And then you
know,

that handles again the long tail.

of any sort of feature from any sort of
app.

that would be yeah, my that's my hope is
that you know,

they they tried to prioritize that,

but yeah, I don't know. It was

So like ideal w ideal

world like MCP for Siri. That's like

Yeah.

kind of ideal and then you know some other
stuff underneath.

Have you played around with visual
intelligence and third party apps?

Because over the last year, I've been
seeing more and more apps show up like
when

I take a screenshot or when I do visual
intelligence.

Like Amazon's in there now. ~ apps like
Bezil have taken advantage of that,

Etsy, where it'll basically do like a
reverse image search.

And you can search

Mm-hmm.

for a product, do g obviously the reverse
Google image search.

But if as a third party app, is there even
an option right now where if

I go to straight to visual intelligence,

could I go to then food noms and send the
picture I just took to Food Noms rather

than Etsy or Amazon? Or is that those
aren't really tools that are built

in yet for you guys?

that's that's another thing that would be
cool.

That would be cool.

Okay.

~ I think you kind of like described it
well in that like I feel like Apple

has designed that feature to be intended
for apps to do something like reverse
image

search in the UX and the UI is like
designed that way in that usually there's
like

more than one result, so they have like
kind of the grid view and

I think there's kind of the expectation
that the results

are processed fairly quickly, like
probably within like half a second or so.

Yeah.

And for me with AI processing,

you know, queries can take generally like
five to ten seconds.

And I don't know, it just kinda seems like
Apple hasn't really quite intended like

certain use cases like you know,

FoodNom's AI ~ with visual intelligence.

But ~ some like small tweaks there would
definitely like I think make that

a lot more yeah like appropriate sort of
use case.

That'd be cool. That'd

be cool. Well, Ryan, thanks so much for
your time.

Food Noms, everyone should go download it.

my wife loves it. I've been use we've both
been using it for the last year.

Pay for premium. It is a beautifully
designed app.

You do incredible work. And the AI stuff
is amazing.

Like I'll take a picture of three street
tacos and it just it gets it.

You know what I mean? And the

Mm-hmm.

results over the past year, I th it's been
very accurate.

And so thanks for your time on this and be
curious to follow will

be following your journey.

if you can implement more series AI in the
future and what Apple does.

So thanks for being here, Ryan.

Yeah, of course. Thanks, Steven.