# Inconsistent syllable error rate between vak eval and predict

**URL:** <https://forum.vocalpy.org/t/inconsistent-syllable-error-rate-between-vak-eval-and-predict/79>\
**Category:** Q&A\
**Tags:** vak\
**Created:** [September 14, 2023, 11:59pm UTC](https://forum.vocalpy.org/t/inconsistent-syllable-error-rate-between-vak-eval-and-predict/79 "2023-09-14T23:59:38Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![zhileizhao](https://avatars.discourse-cdn.com/v4/letter/z/dfb087/32.png) [@zhileizhao](https://forum.vocalpy.org/u/zhileizhao)\
**Post date:** [September 14, 2023, 11:59pm UTC](https://forum.vocalpy.org/t/inconsistent-syllable-error-rate-between-vak-eval-and-predict/79/1 "2023-09-14T23:59:39Z")

</div>

Hi vocalpy team,

Great work on developing the toolkits! Thank you so much for making them publicly accessible!  
I’ve been using vak to train TweetyNet model on my own vocalization data for the annotation task. I’m a little confused about the results. Seems that the syllable accuracy is not correctly calculated by the vak eval function. Here is one example output from vak eval:  
2023-09-14 19:37:19,402 - vak.core.eval - INFO - avg\_acc: 0.85201  
2023-09-14 19:37:19,402 - vak.core.eval - INFO - avg\_acc\_tfm: 0.85201  
2023-09-14 19:37:19,402 - vak.core.eval - INFO - avg\_levenshtein: 45.52941  
2023-09-14 19:37:19,402 - vak.core.eval - INFO - avg\_levenshtein\_tfm: 45.52941  
2023-09-14 19:37:19,402 - vak.core.eval - INFO - avg\_segment\_error\_rate: 0.78289  
2023-09-14 19:37:19,402 - vak.core.eval - INFO - avg\_segment\_error\_rate\_tfm: 0.78289  
2023-09-14 19:37:19,402 - vak.core.eval - INFO - avg\_loss: 0.52653

If I understand the results correctly, my model is able to achieve 85% frame accuracy, but the syllable accuracy is pretty bad. However, if I use vak predict to generate predicted labels, the results weren’t that bad. I compared the predicted labels to the ground-truth labels and calculated the levenshtein distance myself using the metrics.Levenshtein() function, the average syllable error is only 26.8%, instead of 78.2% as claimed by the vak eval function.  
I’ve been thinking about this for a while, but couldn’t figure it out why the syllable error rate is so different between vak predict and eval. Any thoughts?  
I’m using the ‘simple-seq’ format and only have single-char labels.

Thank you!  
Zhilei

---

<div class="post-metadata">

**Author:** ![nicholdav](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.vocalpy.org/nicholdav/32/5_2.png) [@nicholdav](https://forum.vocalpy.org/u/nicholdav)\
**Post date:** [September 15, 2023, 1:07pm UTC](https://forum.vocalpy.org/t/inconsistent-syllable-error-rate-between-vak-eval-and-predict/79/2 "2023-09-15T13:07:59Z")

</div>

Hi @zhileizhao, welcome to the forum, glad to hear you appreciate what we’re doing

> Any thoughts?

Yeah, I think you have found a bug 😬

The metric values with and without the transforms applied should definitely not be the same!

And you have done an excellent job of confirming it by computing the metric yourself, thank you!

Can you please go ahead and raise an issue on our tracker on GitHub reporting this bug?

> **[Build software better, together](https://github.com/vocalpy/vak/issues/new?assignees=&labels=BUG&projects=&template=bug_report.md&title=BUG%3A)**
>
> GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects.

We’ll be happy to give you credit for spotting it. I have a feeling the fix might be pretty simple. If you are feeling up to it I can walk you through contributing a fix to

---

<div class="post-metadata">

**Author:** ![zhileizhao](https://avatars.discourse-cdn.com/v4/letter/z/dfb087/32.png) [@zhileizhao](https://forum.vocalpy.org/u/zhileizhao)\
**Post date:** [September 16, 2023, 7:00pm UTC](https://forum.vocalpy.org/t/inconsistent-syllable-error-rate-between-vak-eval-and-predict/79/3 "2023-09-16T19:00:57Z")

</div>

Hi @nicholdav thank you so much for the quick response!  
Yes, I will raise an issue on the GitHub repo. Glad that I contribute a little bit to this wonderful package!

---

<div class="post-metadata">

**Author:** ![nicholdav](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.vocalpy.org/nicholdav/32/5_2.png) [@nicholdav](https://forum.vocalpy.org/u/nicholdav)\
**Post date:** [September 18, 2023, 8:08pm UTC](https://forum.vocalpy.org/t/inconsistent-syllable-error-rate-between-vak-eval-and-predict/79/4 "2023-09-18T20:08:25Z")

</div>

Excellent, thank you @zhileizhao! I saw you raised the issue, appreciate it. I have it on my to-do list to reply

---

<div class="post-metadata">

**Author:** ![nicholdav](https://yyz2.discourse-cdn.com/flex032/user_avatar/forum.vocalpy.org/nicholdav/32/5_2.png) [@nicholdav](https://forum.vocalpy.org/u/nicholdav)\
**Post date:** [September 20, 2023, 10:11pm UTC](https://forum.vocalpy.org/t/inconsistent-syllable-error-rate-between-vak-eval-and-predict/79/5 "2023-09-20T22:11:41Z")

</div>

Thanks again for raising the issue @zhileizhao.  
I’m just linking it here for future reference: [Inconsistent syllable error rate between vak eval and predict · Issue #697 · vocalpy/vak · GitHub](https://github.com/vocalpy/vak/issues/697)  
Long story short, this was fixed in version 1.0 but I didn’t make note that we needed to fix it in version \< 1.0 – my fault for letting that slip through the cracks.  
A quick fix might be to upgrade to version 1.0 but we should fix the bug in the maintenance version too.

Let’s continue the discussion on GitHub – just let me know how I can help you.  
We’d definitely be happy to help you contribute a fix if you’d like, or I can take care of it.  
Appreciate your catching this bug!