阅读说明：本技术 一种基于音视频演奏示范的交互式乐谱展示方法及系统 (Interactive music score display method and system based on audio and video playing demonstration ) 是由 魏新元 方家文 何欣燕 黄民 于 2021-07-26 设计创作，主要内容包括：本发明公开了一种基于音视频演奏示范的交互式乐谱展示方法及系统,所述方法包括如下过程：将演奏示范文件中的音频和乐谱文件对齐；将对齐结果以光标的形式在乐谱相应的位置上显示出来；演奏示范文件中的音频和乐谱文件对齐的方法如下：获取演奏示范文件中的音频和乐谱文件；将音频文件和乐谱文件进行分帧并逐帧提取特征,逐帧计算每个乐谱帧和音频帧的特征相似度矩阵,取每帧信号与乐谱信号相似度最大值处的坐标值；输出音频和乐谱的对齐路径。本发明提升了用户的交互体验,提高音乐教学与欣赏的效果与便利性。(The invention discloses an interactive music score display method and system based on audio and video playing demonstration, wherein the method comprises the following steps: aligning the audio and score files in the performance demonstration file; displaying the alignment result on the corresponding position of the music score in a cursor form; the method of audio and score file alignment in a performance demonstration file is as follows: acquiring audio and music score files in a performance demonstration file; the audio file and the music score file are subjected to framing, the characteristics are extracted frame by frame, the characteristic similarity matrix of each music score frame and each audio frame is calculated frame by frame, and the coordinate value of the maximum value of the similarity of each frame of signal and the music score signal is taken; and outputting the aligned paths of the audio and the music score. The invention improves the interactive experience of the user and improves the effect and convenience of music teaching and appreciation.)

1. An interactive music score display method based on audio and video performance demonstration is characterized by comprising the following processes:

1) aligning the audio and score files in the performance demonstration file;

2) displaying the alignment result on the corresponding position of the music score in a cursor form;

the method for aligning the audio and music score files in the performance demonstration file comprises the following steps:

11) acquiring audio and music score files in a performance demonstration file;

12) the audio file and the music score file are subjected to frame division and the characteristics are extracted frame by frame,

13) calculating a feature similarity matrix of each music score frame and each audio frame by frame, and taking a coordinate value at the maximum value of the similarity of each frame signal and the music score signal;

14) and outputting the aligned paths of the audio and the music score.

2. The interactive music score presentation method based on audio-video performance demonstration of claim 1, further comprising changing speed and tone of the video and music score of the performance demonstration file, comprising the following steps:

3) changing the speed and tone of the video and the music score of the playing demonstration file;

4) displaying the music score at a corresponding position in a cursor form;

the method for changing the speed of the demonstration video and the music score of the performance demonstration file is as follows:

31) calculating the video image index number corresponding to each audio frame according to the frame rate of the video and the frame length and frame jump when the audio is framed in the performance demonstration file, and realizing sound and picture synchronization;

32) acquiring a speed change multiple, generating a speed-changed audio frame index sequence t 'according to the speed change multiple, and reconstructing an audio time domain signal by using the speed-changed audio frame index sequence t';

33) aligning the reconstructed audio with the music score file to obtain a music score frame index sequence s1 corresponding to the audio frame index sequence after speed change;

34) playing the corresponding image according to the audio frame index sequence after speed change, and simultaneously displaying a cursor at the corresponding position of the music score in real time according to the music score frame index sequence s 1;

the method of transposition of the demonstration video and the music score of the performance demonstration file is as follows:

35) acquiring the pitch score of the rising and falling tone;

36) converting the sound scores into frequencies and expressing the frequencies in a score form;

37) and inputting the time domain audio data and the frequency in the fractional form into a phase vocoder for speed change to obtain an audio signal, and resampling the audio signal after speed change according to the numerator and the denominator of the frequency in the fractional form to obtain the audio after pitch change.

3. The interactive music score display method based on audio-video performance demonstration as claimed in claim 2, wherein the step of generating the audio frame index sequence t' after speed change according to the speed change multiple comprises the specific steps of:

the variable speed multiple a of the input audio, the original audio frame sequence t is 0, 1, 2 …, N, N is the total frame number of the original audio, after the variable speed a times, the new audio frame sequence becomes t ', wherein t' is 0, a, 2a, 3a … N;

the specific steps of reconstructing the audio time domain signal by using the audio frame index sequence t' after the speed change are as follows:

any element m in the audio frame index sequence t' is rounded down to obtain an integer frame which is n, and the decimal part alpha is between the nth frame and the n +1 frame of the original audio;

reconstructing the amplitude spectrum of the mth frame in the audio frame index sequence t' to obtain:

S_m＝(1-α)S_n+αS_n+1

and calculating the phase spectrum of the mth frame to obtain:

wherein S_mAnd P_mThe mth element corresponds to the amplitude spectrum and the phase spectrum of the frame respectively,indicating the phase increment, P, corresponding to the current m-th element_t(n-1)Representing the phase corresponding to the (n-1) th frame;

fourier transform C after reconstruction of mth frame signal_mExpressed as:

C_m＝S_m*exp(i*P_m)

wherein i is a plurality of C_mThat is, the mth element in the audio index sequence t' after the speed change, that is, the frequency domain signal corresponding to the mth frame, and the time domain signal of the mth frame can be obtained after the frequency domain signal is subjected to inverse fourier transform.

4. The interactive music score presentation method based on audio-video performance demonstration of claim 2, wherein the formula of converting the sound score into frequency is as follows:

wherein m represents the sound score;

the formula for the frequency to be expressed in a fractional manner is:

wherein the numerator and denominator are respectively a₁And a₂；

The formula for resampling the audio signal is:

s_shift＝resample(s_pv,a₁,a₂)

wherein s is_pvRepresenting an audio signal, s_shiftRepresenting the transposed audio.

5. The method for displaying an interactive music score based on an audio-video performance demonstration of claim 1, further comprising a method for merging a plurality of audio-video demonstration files, comprising the following steps:

5) merging the plurality of performance demonstration files;

6) displaying the music score at a corresponding position in a cursor form;

the method for merging a plurality of performance demonstration files comprises the following steps:

51) shifting the corresponding sound part of the audio frequency in each performance demonstration file to the same tone as the general spectrum;

52) selecting the A mode or the B mode for sound part combination:

wherein, the mode A: aligning all demonstration audios with the general spectrums by using a file alignment module, changing the speed of the audios of all performance demonstration files according to an alignment path to ensure that the speeds of all the performance demonstration files are uniform to be the same as the general spectrums, and then combining the speed-changed performance demonstration files;

mode B: selecting one of the performance demonstrations as a reference demonstration, aligning the reference demonstration with the score, modifying the score according to an alignment path to match the score with the performance of the reference demonstration, aligning the modified score with other performance demonstrations, shifting the audio frequencies of all the performance demonstration files except the reference demonstration according to the alignment path, and merging the performance demonstration files after shifting.

6. An interactive music score presentation system based on audio-video performance demonstration is characterized by comprising:

the system comprises a file alignment module, a play operation module and a performance demonstration file synthesis module, wherein the file alignment module is used for aligning the audio and the music score path in the performance demonstration file;

the playing operation module is used for carrying out interactive synchronous playing, speed changing and tone changing on the video and the music score in the playing demonstration file;

the performance demonstration file synthesis module is used for merging a plurality of performance demonstration files.

7. The system of claim 6, further comprising a labeling module for labeling the music score and inserting multimedia files.

8. The interactive music score presentation method based on audio-video performance demonstration of claim 7, wherein the labeling module is implemented by the steps of:

the acquisition instruction opens a labeling function, notes, measures, phrases and paragraphs to be labeled are selected on the music score, an index range is calculated, and then a multimedia file is embedded in the selected music score index range based on a multimedia label technology of html 5.

9. The system of claim 8, wherein the annotations in the music score are synchronously uploaded to a remote server, and when other users request the current music score, the annotations in the current music score are synchronously transmitted to the users.

10. The system of claim 8, wherein the multimedia file comprises: drawing, text, picture, audio and video.

Technical Field

The invention relates to the technical field of music score interactive display, in particular to an interactive music score display method and system based on audio and video playing demonstration.

Background

With the increasing popularization of music education and the increasing development of internet industry, audio and video playing/teaching and other materials related to music are more and more, and especially the playing audio and video demonstration of teachers/musicians greatly meets the audio and video/teaching requirements of broad music lovers/learners.

In the context of music teaching or appreciation, music scores are often relied upon. However, these audio and video demonstrations do not usually include a music score, and the common operations of labeling on the music score, locating the position of the currently played demonstration on the music score and the like still need to be performed in a conventional manner, and on this basis, operations such as speed changing, tone changing and the like cannot be supported, which is very inconvenient.

In the prior art, the publication number is CN109377818A, and chinese invention patent discloses a music score playing module assembly of a digital music teaching system in 2019, 2.22.2019, wherein the music score playing module assembly includes a music score playing unit, a roll playing unit, a model singing unit, an accompaniment playing unit and a lyric playing unit. The music score playing module component of the digital music teaching system can provide a plurality of playing modes, can meet the requirements of users, can play recorded songs and accompaniment, can identify and automatically synthesize and play the written music score, can meet the teaching requirements of teachers, and can meet the use requirements of students; in the playing process, the positions of music symbols on the keyboard are displayed by simulating the music keyboard, and the one-to-one mapping of the musical notes, the lyrics and the virtual keyboard is realized. The scheme realizes the playing of the music score and does not realize interactive functions such as tone variation, speed change and the like.

Disclosure of Invention

The invention provides an interactive synchronous music score display method and system based on audio and video playing demonstration, aiming at overcoming the defect that the conventional music score display cannot realize user-friendly interaction.

The primary objective of the present invention is to solve the above technical problems, and the technical solution of the present invention is as follows:

the invention provides an interactive music score display method based on audio and video performance demonstration in a first aspect, which comprises the following processes:

1) aligning the audio and score files in the performance demonstration file;

2) displaying the alignment result on the corresponding position of the music score in a cursor form;

the method for aligning the audio and music score files in the performance demonstration file comprises the following steps:

11) acquiring audio and music score files in a performance demonstration file;

12) the audio file and the music score file are subjected to frame division and the characteristics are extracted frame by frame,

13) calculating a feature similarity matrix of each music score frame and each audio frame by frame, and taking a coordinate value at the maximum value of the similarity of each frame signal and the music score signal;

14) and outputting the aligned paths of the audio and the music score.

Furthermore, the method also comprises the steps of changing the speed and the tone of the video and the music score of the playing demonstration file, and comprises the following steps:

3) changing the speed and tone of the video and the music score of the playing demonstration file;

4) and displaying the music score at a corresponding position in a cursor form.

The method for changing the speed of the demonstration video and the music score of the performance demonstration file is as follows:

31) calculating the video image index number corresponding to each audio frame according to the frame rate of the video and the frame length and frame jump when the audio is framed in the performance demonstration file, and realizing sound and picture synchronization;

32) acquiring a speed change multiple, generating a speed-changed audio frame index sequence t 'according to the speed change multiple, and reconstructing an audio time domain signal by using the speed-changed audio frame index sequence t';

33) aligning the reconstructed audio with the music score file to obtain a music score frame index sequence s1 corresponding to the audio frame index sequence after speed change;

34) playing the corresponding image according to the audio frame index sequence after speed change, and simultaneously displaying a cursor at the corresponding position of the music score in real time according to the music score frame index sequence s 1;

the method of transposition of the demonstration video and the music score of the performance demonstration file is as follows:

35) acquiring the pitch score of the rising and falling tone;

36) converting the sound scores into frequencies and expressing the frequencies in a score form;

37) and inputting the time domain audio data and the frequency in the fractional form into a phase vocoder for speed change to obtain an audio signal, and resampling the audio signal after speed change according to the numerator and the denominator of the frequency in the fractional form to obtain the audio after pitch change.

Further, the specific steps of generating the audio frame index sequence t' after the speed change according to the speed change multiple are as follows:

the variable speed multiple a of the input audio, the original audio frame sequence t is 0, 1, 2 …, N, N is the total frame number of the original audio, after the variable speed a times, the new audio frame sequence becomes t ', wherein t' is 0, a, 2a, 3a … N;

the specific steps of reconstructing the audio time domain signal by using the audio frame index sequence t' after the speed change are as follows:

any element m in the audio frame index sequence t' is rounded down to obtain an integer frame which is n, and the decimal part alpha is between the nth frame and the n +1 frame of the original audio;

reconstructing the amplitude spectrum of the mth frame in the audio frame index sequence t' to obtain:

S_m＝(1-α)S_n+αS_n+1

and calculating the phase spectrum of the mth frame to obtain:

wherein S_mAnd P_mRespectively m th elementCorresponding to the magnitude spectrum and phase spectrum of the frame,indicating the phase increment, P, corresponding to the current m-th element_t(n-1)Representing the phase corresponding to the (n-1) th frame;

fourier transform C after reconstruction of mth frame signal_mExpressed as:

C_m＝S_m*exp(i*P_m)

wherein i is a plurality of C_mThat is, the mth element in the audio index sequence t' after the speed change, that is, the frequency domain signal corresponding to the mth frame, and the time domain signal of the mth frame can be obtained after the frequency domain signal is subjected to inverse fourier transform.

Further, the formula for converting the sound score into frequency is as follows:

wherein m represents the sound score;

the formula for the frequency to be expressed in a fractional manner is:

wherein the numerator and denominator are respectively a₁And a₂；

The formula for resampling the audio signal is:

s_shift＝resample(s_pv,a₁,a₂)

wherein s is_pvRepresenting an audio signal, s_shiftRepresenting the transposed audio.

Further, the method for merging the plurality of audio and video demonstration files comprises the following steps:

5) merging the plurality of performance demonstration files;

6) displaying the music score at a corresponding position in a cursor form;

the method for merging a plurality of performance demonstration files comprises the following steps:

51) shifting the corresponding sound part of the audio frequency in each performance demonstration file to the same tone as the general spectrum;

52) selecting the A mode or the B mode for sound part combination:

wherein, the mode A: aligning all demonstration audios with the general spectrums by using a file alignment module, changing the speed of the audios of all performance demonstration files according to an alignment path to ensure that the speeds of all the performance demonstration files are uniform to be the same as the general spectrums, and then combining the speed-changed performance demonstration files;

mode B: selecting one of the performance demonstrations as a reference demonstration, aligning the reference demonstration with the score, modifying the score according to an alignment path to match the score with the performance of the reference demonstration, aligning the modified score with other performance demonstrations, shifting the audio frequencies of all the performance demonstration files except the reference demonstration according to the alignment path, and merging the performance demonstration files after shifting.

The invention provides an interactive music score display system based on audio and video performance demonstration, which comprises:

the system comprises a file alignment module, a play operation module and a performance demonstration file synthesis module, wherein the file alignment module is used for aligning the audio and the music score path in the performance demonstration file;

the playing operation module is used for carrying out interactive synchronous playing, speed changing and tone changing on the video and the music score in the playing demonstration file;

the performance demonstration file synthesis module is used for merging a plurality of performance demonstration files.

Furthermore, the system also comprises a labeling module, and the labeling module is used for labeling the music score and inserting the multimedia file.

Further, the labeling module comprises the following implementation steps:

the acquisition instruction opens a labeling function, notes, measures, phrases and paragraphs to be labeled are selected on the music score, an index range is calculated, and then a multimedia file is embedded in the selected music score index range based on a multimedia label technology of html 5.

Furthermore, the labels in the music score are synchronously uploaded to a remote server, and when other users request the current music score, the labels carried by the current music score are synchronously sent to the users.

Further, the multimedia file comprises: drawing, text, picture, audio and video.

Compared with the prior art, the technical scheme of the invention has the beneficial effects that:

according to the interactive music score display method based on the audio and video playing demonstration, through alignment of the playing demonstration file and the music score file, interactive tone changing and speed changing and multi-file synthesis, the interactive experience of a user is improved, and the effect and convenience of music teaching and appreciation are improved.

Drawings

Fig. 1 is a flowchart of an audio and score file alignment method in a performance demonstration file according to the present invention.

Fig. 2 is a flowchart of a method for changing the speed of an exemplary video and music score playing an exemplary file according to the present invention.

Fig. 3 is a flowchart of a method for tonal modification of an exemplary video and score playing an exemplary document in accordance with the present invention.

Fig. 4 is a block diagram of a first exemplary interactive music score presentation system based on audio-video performance according to an embodiment of the present invention.

Fig. 5 is a block diagram of a second exemplary interactive music score presentation system based on audio-video performance according to an embodiment of the present invention.

Detailed Description

In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.

Example 1

An interactive music score display method based on audio and video performance demonstration comprises the following processes:

1) aligning the audio and score files in the performance demonstration file;

2) displaying the alignment result on the corresponding position of the music score in a cursor form;

it should be noted that, the audio and music score files in the performance demonstration file are aligned, as shown in fig. 1, the steps are:

11) acquiring audio and music score files in a performance demonstration file;

12) the audio file and the music score file are subjected to framing and the characteristics are extracted frame by frame;

13) calculating a feature similarity matrix of each music score frame and each audio frame by frame, and taking a coordinate value at the maximum value of the similarity of each frame signal and the music score signal;

14) and outputting the aligned paths of the audio and the music score.

It should be noted that the output alignment path, i.e. the current audio frame, corresponds to a specific frame on the score. If the error of the alignment result is large, the correction can be carried out manually. The final result is displayed on the score in real time in the form of a cursor.

It should be noted that, in the present invention, all operations of matching audio and video playing positions in the music score and the playing demonstration file are implemented by the above-mentioned audio and music score alignment result, for example, dragging the progress to jump to a certain position on the playing demonstration file can determine the number of frames where the audio is located, and according to the alignment result, the music score frame corresponding to the frame of audio can be found, and then displayed on the corresponding position of the music score in the form of a cursor. And vice versa, namely, selecting a position on the music score, and jumping the progress bar of the performance demonstration file to the corresponding position for playing. Including performing the above operations on an audiovisual demonstration that has been shifted or transposed.

In the scheme, the method also comprises the steps of changing the speed and the tone of the video and the music score of the playing demonstration file, and comprises the following steps:

3) changing the speed and tone of the video and the music score of the playing demonstration file;

4) and displaying the music score at a corresponding position in a cursor form.

In the present invention, as shown in fig. 2, a method of changing speed of an exemplary video and a score of a performance exemplary file is as follows: :

31) calculating the video image index number corresponding to each audio frame according to the frame rate of the video and the frame length and frame jump when the audio is framed in the playing demonstration file, and realizing sound and picture synchronization;

32) acquiring a speed change multiple, generating a speed-changed audio frame index sequence t 'according to the speed change multiple, and reconstructing an audio time domain signal by using the speed-changed audio frame index sequence t';

33) aligning the reconstructed audio with the music score file to obtain a music score frame index sequence s1 corresponding to the audio frame index sequence after speed change;

34) and playing the corresponding image according to the audio frame index sequence after the speed change, and simultaneously displaying the cursor at the corresponding position of the music score in real time according to the music score frame index sequence s 1.

In the invention, the specific steps of generating the audio frame index sequence t' after speed change according to the speed change multiple are as follows:

the variable speed multiple a of the input audio, the original audio frame sequence t is 0, 1, 2 …, N is the total number of original audio frames, and for convenience, 0: 1: n, after a multiple of a shift, the new sequence of audio frames becomes t ', where t' is 0, a, 2a, 3a … N, and for convenience can be represented by 0: a: n represents;

for example, if the total length of the current audio is N frames, the sequence t of all audio frame indexes of the original audio is an integer from 1 to N, when the playing speed is changed to 2 times, the audio frame index t is only changed to 0, 2, 4 … N, and then the music score frame index sequence s1 corresponding to the audio frame index is selected according to the alignment path of the first step;

the method for reconstructing the audio time domain signal by using the audio frame index sequence t' after the speed change comprises the following specific steps:

any element m in the audio frame index sequence t' is rounded down to obtain an integer frame which is n, and the decimal part alpha is between the nth frame and the n +1 frame of the original audio;

reconstructing the amplitude spectrum of the mth frame in the audio frame index sequence t' to obtain:

S_m＝(1-α)S_n+αS_n+1

and calculating the phase spectrum of the mth frame to obtain:

wherein S_mAnd P_mThe mth element corresponds to the amplitude spectrum and the phase spectrum of the frame respectively,represents the current mth element

Phase increment, P, corresponding to the element_t(n-1)Representing the phase corresponding to the (n-1) th frame;

fourier transform C after reconstruction of mth frame signal_mExpressed as:

C_m＝S_m*exp(i*P_m)

wherein i is a plurality of C_nThat is, the mth element in the audio frame index sequence t' after the speed change, i.e., the frequency domain signal corresponding to the mth frame, is obtained after the frequency domain signal is subjected to inverse fourier transform.

Further, as shown in fig. 3, a method of transposing an exemplary video and a score of a performance exemplary file is as follows:

35) acquiring the pitch number of the rising and falling tone;

36) converting the sound scores into frequencies and expressing the frequencies in a score form;

the formula for converting the voice score into frequency is as follows:

wherein m represents the sound score;

the formula for the frequency to be expressed in a fractional manner is:

wherein the numerator and denominator are respectively a₁And a₂；

It should be noted that the present invention converts rising and falling scores of tones into frequencies according to the twelve-tone equal temperament.

37) The time domain audio data and the frequency in the fractional form are input into a phase vocoder to obtain an audio signal after speed change, and the audio signal after speed change is resampled according to the numerator and the denominator of the frequency in the fractional form to obtain the audio after tone change. The formula for resampling the audio signal is:

s_shift＝resample(s_pv,a₁,a₂)

wherein s is_pvRepresenting an audio signal, s_shiftRepresenting the transposed audio.

It should be noted that the modified audio frequency s_shiftIs substantially the same as the length of the original audio, the main error being to convert the frequency f of the panning from a fraction of a small number to a fractionAt the moment of introduction.

In a specific embodiment, after the audio frequency is subjected to speed change and tone change, the audio frequency subjected to tone change processing is aligned with a corresponding music score, so that the function that any note on the music score is clicked by the audio frequency subjected to speed change, a progress bar for playing the audio and video of a demonstration file automatically jumps to a corresponding position for playing is realized, and conversely, a video progress bar for playing the demonstration file is dragged to a certain position, and a cursor on the music score immediately jumps to the corresponding note.

Further, the method for merging the plurality of audio and video demonstration files comprises the following steps:

5) merging the plurality of performance demonstration files;

6) displaying the music score at a corresponding position in a cursor form;

the method for merging a plurality of performance demonstration files comprises the following steps:

51) shifting the corresponding sound part of the audio frequency in each performance demonstration file to the same tone as the general spectrum; the total spectrum is a spectrum with all the vocal parts combined together, and all the vocal part velocities are uniform;

52) selecting the A mode or the B mode for sound part combination:

mode A: aligning all demonstration audios with the total spectrum, then performing variable speed expansion, unifying the speeds of all demonstration files to be the same as the total spectrum, and then merging;

mode B: selecting one of the performance demonstrations as a reference demonstration, aligning the reference demonstration with the score, modifying the score according to an alignment path to match the score with the performance of the reference demonstration, aligning the modified score with other performance demonstrations, performing variable speed expansion and contraction, and finally combining all the vocal parts.

It should be noted that, said merging of multiple performance demonstration files is to merge the corresponding sound parts of multiple performance demonstration files, and the sound parts may be different tones and arbitrary free performance speeds, wherein, although the mode a can merge all the performance demonstration files neatly, the disadvantage is that there is no emotional color of personal performance, and the emotional color is played mechanically at a speed unified by the music score, and the aesthetic feeling is poor; mode B is that the combined audio will be played according to the rhythm in the reference demonstration, rather than the fixed rhythm in mode a, which is more aesthetic.

Fig. 4 shows a block diagram of a first interactive music score presentation system based on an audio-video performance demonstration.

The invention provides an interactive music score display system based on audio-video performance demonstration, which is characterized by comprising:

the system comprises a file alignment module, a play operation module and a performance demonstration file synthesis module, wherein the file alignment module is used for aligning the audio and the music score path in the performance demonstration file;

the playing operation module is used for carrying out interactive synchronous playing, speed changing and tone changing on the video and the music score in the playing demonstration file;

the performance demonstration file synthesis module is used for merging a plurality of performance demonstration files.

Fig. 5 shows a second interactive music score presentation system block diagram based on an audio-video performance demonstration.

The system also comprises the labeling module, and the labeling module is used for labeling the music score and inserting a multimedia file;

the marking module comprises the following implementation steps:

the acquisition instruction opens a labeling function, notes, measures, phrases and paragraphs to be labeled are selected on the music score, an index range is calculated, and then a multimedia file is embedded in the selected music score index range based on a multimedia label technology of html 5. The multimedia file comprises: drawing, text, picture, audio and video.

It should be noted that, in a specific embodiment, the operations of labeling and displaying the music score can be performed in an svg (scalable vector graphics) framework, and in the present invention, all operations of drawing at any position of the music score, and inserting multimedia files such as drawing, text, picture, audio/video, etc. into any note, measure, phrase, paragraph of the music score are implemented by a labeling module.

The annotations in the music score are synchronously uploaded to a remote server, and when other users (such as students of the same teacher) request the current music score, the annotations carried by the current music score are synchronously sent to the users.

The playing operation module is used for carrying out interactive synchronous playing, speed changing and tone changing on the video and the music score in the playing demonstration file;

it should be noted that, when the interactive synchronous playing is performed, that is, any note on the music score is clicked, the video progress bar jumps to the position of response for playing; and vice versa, the video progress bar is dragged to a certain position, and the cursor on the music score immediately jumps to the corresponding note.

It should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.