Manuela
So, hi, everyone. Great to see you here. It's nice to be in Paris and nice to talk about the Deep Dive into Semantics widget with you. So my name is Manuela as you have heard and I'm from Berlin. You can find me on Blue Sky or on LinkedIn. And I'm currently working as a freelancer Flutter developer focusing on accessibility. I'm also co-organizing the Flutter Berlin meetup with this nice gentleman before me, Sally, and other nice people. And I'm also part of the Flutteristas community. So if you are a female or a non-binary Flutter developer, feel free to join. So, let's start here with this. This is an app as we see it as sighted users. We have a clean UI, we have an organized layout and nice colors. Now a screen reader user might be only seeing this, nothing, and there's just silence, no button, no interaction and no way to interact. When we flip the switch, here's the semantics annotated on the Flutter app. And we see suddenly the whole app looks very, very different. That difference, that's the semantic system. And this is what the talk is about. So first of all, we have to talk about people. And WHO said 15% of people are significantly disabled in this world. That's one in seven people, which is a huge amount of people if you think about that. So accessibility is not just a bonus for those people. It's not just, let's put it at the end of the roadmap. It's essential. They cannot use your app without it. And ignoring it means ignoring a large user base, which is quite unfortunate because as developers, we want our app to be used by many people. So today's focus is the semantics widget, which is a key to have inclusive Flutter apps. So, maybe you have seen this EAA, which means European Accessibility Act. This mandates that all digital services need to be accessible. There are of course some exclusions, but if you have some services and you sell something, you have to follow it and it comes into act in June 2025. So you have two months left to make stuff accessible. Have fun. And for EU markets, that means accessibility is not an afterthought. It's required. And yeah, even without laws, I think we want to be nice developers and build nice products that, you know, can be used by everyone. So let's talk about how Semantics Widget helps us to get there. And yeah, first of all, we have to probably establish what are semantics in Flutter. So we know why accessibility matters now, which brings us to Semantics Widget in Flutter. So you can think of semantics being a bridge between Flutter UI and platform accessibility services. What does that mean? It connects the UI with tools like VoiceOver or TalkBack on Android. And without it, screen readers can't describe your app meaningfully, which is quite unfortunate for those users if you don't have semantics. And it's basically what makes a button be a button and not just like random pixels on the screen. And yeah, let's break down this bridge and see how it actually works under the hood, which is maybe the interesting part. So we as Flutter developers, we build Flutter widgets and we think mostly of the widget tree. But Flutter has many, many more trees. It's basically like a forest. So when we have a widget tree, it's the declarative tree. Then we have element tree, which handled some lifecycle and some state. and the rendering tree has some like layouting and painting the widgets on our screen. And the rendering tree has also semantics information, which then in turn creates the semantics tree. So when we look at our normal Flutter counter app, we will see the simple UI. There's a button, there's a text, there's a counter. And we as sighted users instantly see what's happening. We know we have to tap on that button and the counter increases, but a screen reader cannot see the UI as we can see it, right? So it depends entirely on the semantics tree. And each meaningful widget should become a semantic node and have some semantic information for the semantic stream. And Flutter's job there is to expose those nodes, to label them correctly and structure them for accessibility services. So when we think in terms of the widget tree, because we as frontend developers probably think more of the UI, right? So we build like a text, a button and so on. But Flutter doesn't use native UI components. It basically renders everything on a canvas. So the screen readers of the native platforms cannot detect, okay, that's a button. And that's why you have to explicitly describe everything that has a meaning to the screen readers so they understand what's happening there. So in a sense, we would say, okay, this is a button, this is a heading, and this is an image with a description. And it's basically what makes Flutter usable for everyone. So now let's get a little bit deeper into all of that. So I have talked about the render tree and each widget has a render object for layout and painting. But this render object has also semantics information in the semantics configuration. And this basically acts a little bit like a resume for accessibility. It has information like, hey, you know what, I'm a button, my label is play, I'm currently disabled, but I can be tapped. So there are many informations that might be tied to that widget. And that includes, as I said, roles, actions, labels, and states. And Flutter uses this info to decide which nodes to add to the semantics tree and what property each node gets. So, once we have all those semantic configurations, we can finally build the semantics tree. It skips purely visual or structural widgets like containers or paddings. We don't need information about that. And only includes widgets with meaningful accessibility info. And it's way more streamlined than the widget tree, as you can see. So the semantics tree is basically a visual representation of UI that's very, very basic. And this is what's sent to the OS, to the accessibility service. So now that we have our semantics tree built, let's manage it. And the manager of the tree is the semantics owner. Basically, it keeps the tree in memory and it tracks which node has changed and then updates the tree. that whole thing. So basically, you have a semantics update builder, which collects the updated nodes and creates a semantics update, which then will be sent later on through the Flutter engine to the native API. And that makes the system very efficient and only sends update when something meaningful changes. So I told you about the semantics owner being like a manager. So basically it's constantly asking what changed and what can I update? And it doesn't rebuild the tree for each frame. It's just patches the parts that have like changed and sends the updates. So, or prepares the update because sending the updates is the role of the semantics binding, which we will talk about now, which acts like a bridge between the Flutter UI and the engine. And it basically checks if accessibility is enabled first. And if you have turned on accessibility services for Flutter web, you will notice that there's a button accessibility button. So it only will. build it all when you have accessibility service actually enabled. This is easily to find out on mobile, but on browsers we have no way. That's why there is this button. So basically, if it's enabled, it pushes the updates to the engine and it listens to feedbacks from the OS. So let's say you're navigating your app with the screen reader and you tap a button. this button has some kind of callback, right? So the semantics binding checks, okay, which button was pressed and which is the callback that we should actually enable. All right, and now we're already at the part where we go to the native side. So from the Flutter engine, the semantic update will be translated to native APIs. For iOS is UI accessibility, for Android it's accessibility node info, for Microsoft it's iAccessible. And for web we have then ARIA labels and custom DOM, which is created. And this is how a Flutter button can be finally seen as a button for TalkBack or for like VoiceOver or any other assistive technology, which is amazing. So, let's recap. First we create widgets. Then we have render objects with the widgets, right? They describe semantic config, so we have all the information. Then the semantics tree is built and the semantic owner kind of checks, is something changed? Did anything change? Do I have to update something? And then semantic binding sends the update to Flutter Engine, which then translates everything to native APIs. So this was all very, very theoretical and it's probably not something that you will necessarily need in your day to day basis, but you probably need to know how to use everything. So if you look at semantic widgets in the docs, you see you have a lot of parameters, many, many things that you can add. And it makes sense because the screen reader depends on that assistive technology depends on all those little information. I have created some parameter categories. They are not official. They're just like my thing because I like to categorize stuff sometimes. And I made a general categorization for stuff that's like kind of basic that didn't fit into any other more specialized. parameters. We have descriptive parameters like labels, where you give some description, something for states where you just give a state that something is enabled or checked or selected. You can also give a type for very custom widgets like a button or a text field, an image. Then about focus, if something is focusable or if something has focus or what should happen when this widget gets focus. Then we have something about navigation. We have some parameters on callbacks and a lot for text inputs because the screen reader can do a lot with text inputs. There are a lot of shortcuts. that you can take and there it's very important to use the right parameters. If you have a custom text input and then there's special stuff like custom semantics action, which you would use when you have like a very custom UI and something, a custom action should happen. So for instance, when you have a slide to delete or something like that. Let's start with container because I'm only going to talk about the ones that I printed fat in that other slide. So a container decides whether a note is added to the semantic stream. So it's basically checks. if you have to introduce a new dedicated semantics node or by default it's false, the semantics are merged with the parents. And the use case is that basically when you have a reusable component, you might want to use it or when you want to wrap multiple elements into one parent node. Then we have explicit child nodes. So explicit child nodes basically controls how child semantics behave. So if it's set on true, each child must create their own explicit semantic nodes to contribute to semantics. And if it's set on false, children contribute to their parent nodes semantics. So it basically is kind of read together. And it's usually useful for granular control if you want to, you know, control like some little navigation and each child should be read by itself. And it prevents accidental merging of those nodes together. All right, now we have exclude semantics, which is not to be confused with the exclude semantics widget. So there are some properties and some widgets that kind of sound like they're doing the same things, but they're not. So we have exclude semantics, which basically ignores all child semantic nodes, which means if you wrap a semantics around some widgets that you want to exclude. you want to exclude the semantics from, you can give it like a custom label. So it's basically saying, I don't care what the children are saying, please listen to what I have to say. And exclude semantic widgets is just excluding everything. So you're wrapping it around something and it basically just drops it all. And block semantics, on the other hand, is used for like a pop-up. So basically, if you would have like a pop-up and you have content behind the pop-up so that the focus is not moving behind the pop-up, if you know what I mean. So this would make the screen reader user struggle what's actually happening on the screen and that shouldn't happen. So it's very easy. So if it's true, it ignores all the child semantic nodes completely and false. Basically, that's the default. They are still included in the semantics tree and read normally. All right, now we have label, which is probably more of a basic one. It's very descriptive. It's just a text that you would give your semantics node and that will be read out. Yeah, so there it's a little bit different and you have to always understand with labels and with any semantics information that you give, people don't see what's happening on the screen. So basically, if you have a lot of numbers or anything that's kind of like scattered around, try to close your eyes and just listen to what the screen reader is saying, because sometimes you don't get the context and you maybe need some context to it. Then we have live region, which is also quite handy. So whenever you have something in the app appearing that updates, you want to read it out without maybe shifting the focus. Because one thing that you shouldn't do is just manually shifting the focus of the user, because that's confusing. Because screen reader users have a specific way of using the app and the OS is usually handling it. So don't mess with their focus, please. That's why we have live region, which will read out anything that updated. And on iOS and Android, it will generate a polite announcement. That means it's non-interruptive. So you see the focus didn't shift. If you saw the little border around the widget, it doesn't shift, which is a nice thing that you can use within your app. Sometimes it also makes sense to have some semantic announcement when you have a like button because the screen reader user won't see the filled like button, right? So you have to announce it. There's also a thing that you can use like semantics announcement that will handle that. Then you have an identifier. This is probably what a lot of people maybe already know because it's used for tests. It's for widget tests or like Appium tests to reliably identify specific things in your app like a back button, right? So that is not read out by the screen reader, so don't worry if you see some unlocalized string in the identifier, because it's not seen from the screen reader, it won't be read out. Then we do have the sort key, which basically controls the order in which semantics nodes are read out. So basically when you have some custom UI and the screen reader can't use its own logic, you might want to give your widgets an ordinal sort key to basically have a value that will determine which widget should be read out first. So you see, you can customize it very much. I would use it personally very carefully because usually the screen reader reads from the top left and goes down. And this is the logical order it kind of expects. So I wouldn't mess with that if you don't have to. So this one is probably very nice when you have some cards or when you have some forms or something like that to handle that better. Now we get to the last parameters which have to do with navigation. We have scopes route, which basically marks the semantics or marks for the accessibility services that a new section begins. So when you have a pop-up or when you have a new screen or a dialogue, then you would use scopes root. Alone, it won't do anything. Alone, it will only mark that you have basically a new section. But in combination with names route, it will actually read out on which screen you are. So it will read out the label of its child. So you would maybe give it a screen name. So in my case, maybe a checkout page. And that will be read out. Maybe you have heard it when it announced age confirmation dialogue. So that is what is read out and what's maybe helpful for the user because the first element on which you land is the close button. So the user does something, suddenly this pop up comes up and here's first close button, which might be not the best UX. So that's when you can, for instance, use it. So, let's go back to where we started. We started out with like a very empty screen when we don't have any semantic nodes. And now we have like a very filled node, like filled app screen with semantic nodes. And that's great because you have buttons, you have headers and every screen speaks. If you use those semantics wisely. And I say use them wisely because they can, of course, have some performance impact if you just put them everywhere. So use them with a lot of, yeah, use them purposefully. That is what I wanted to say. So, yeah, let's make every screen speak. Thank you so much for listening to me and have a great conference day.
Vadym
Yeah. And still, we have pretty cute questions here. And one I really liked, are sort keys inverted when RTL is on? Do you know that?
Manuela
I have no idea. I have no idea.
Vadym
We have no idea. We have to just try, like, no, I unfortunately just worked with like European apps before, so like...
Vadym
Oh, yeah, that's another area. I think we can ask Ruf actually. Ruf can help with RTL.
Vadym
And then some questions regarding probably your opinion, but who should be responsible for adding and defining semantics label? Product owners, designers, devs, someone else?
Manuela
It's a work that you have to do all together because... Usually you have already an app. Usually it's already written, so it's an afterthought. So what I like to do is to go over every screen and if I can, usually the labels are kind of already added in the app because you have already localization. So most of the time you don't really need any input from designers or product owners. But sometimes you do, so it's like a situative thing. So if something is very custom, I would ask the designers and maybe a PO just to not change a company voice. So yeah, I would say it depends. So if it's an easy thing, I would just edit myself.
Vadym
Thank you. One question from me, by the way. Which one I wanted to ask you. I had a few, so like probably like, do you still cover semantics with tests? You personally.
Manuela
So, well, it depends. I'm a freelancer, so I have to do what the client wants. But I would suggest to have at least some tests for main flows. For accessibility, they're kind of... like not difficult to write but annoying to write because you have to be very exact with everything but for the main flows I think it's worth it to just check that nothing broke currently like I'm even thinking maybe one could have some kind of AI because you can get the semantic nodes right if you use debug dump semantic trees, so you get all the nodes, maybe you could generate tests based on those nodes. But then you have to also manually test before the app if everything is working. And I haven't tried this approach yet. So maybe it works, maybe not.
Vadym
Yeah, yeah. And then another question for me, by the way, is... Have you had experience working with a person who can actually work with the application? So to highlight what are the problems of it in the sense of semantics or like accessibility things?
Manuela
You mean like an accessibility consultant? Or like a person who really have a need?
Vadym
Yeah, so a user with some disabilities.
Manuela
So I personally have interviewed a blind person for one of my talks. And he told me about his experiences with apps and was kind of like heart wrenching because like he really wanted to use the apps and he couldn't sometimes.
Vadym
Yeah, I remember I've seen that. I've seen that.
Manuela
So I would probably recommend to have like also like disabled users take part of user testing. and you can reach out to them over like, you know, their disability support groups. And usually they're super happy to help. So, and of course you pay them, but like usually they're super happy to even hear from tech companies because they want to use your product. They're also sad if they can't, so.
Vadym
Yeah, we had the same experience actually. We had accessibility check of our application and we were really surprised how inaccessible it was. Now it is, of course, much better at least. Yeah, that's something I would encourage everyone to do. It's important. It's really important. And people are really happy to know that we think of them. So they are not abandoned. It's all about human beings and connections. And then test semantics, we checked. Can we use AI to automatically generate the semantic labels?
Manuela
I mean, you can, but do they work is the question because a lot of work when you work with accessibility is manually testing because like there is no way around really manually checking is it read correctly and then you have like sometimes differences between flutter web and maybe mobile or between android and ios so there's no way around manually testing your app and this is where i haven't found a way to use ai in a smart way because Yeah, this is the most time consuming part of accessibility.
Vadym
Yeah, and writing it right, not just to write it. There are some other questions about keyboard traversal, but it's more about focus things, not the semantics. Um, quite a sum there, but let me pick probably the last one. How can you implement translations in your semantic system?
Manuela
Oh, you just can use like localization and just use localized labels. And then that's important, by the way, like a lot of people just dump like English labels in, but of course they need to be localized as well. That would be great. And also if you have like a part of your app, there's like also this attributed property in semantics widget where you can, for instance, say, okay, read this part like with German screen reader and then switch to French, which is also super helpful for like, I don't know, if you have language selector or I don't know, a language app, maybe that makes sense to use as well.
Vadym
Yeah, just like to highlight, it's pretty wide topic, really wide. It's like much more than just one widget. There are three widgets, but it's better to use only one or two and better not to over complicate as well. But I think that's pretty, at least at this point, I'm encouraging you to ask questions afterwards. Amazing Manuela can answer most of them at least.
Manuela
I hope so.
Vadym
Thank you very much.