Andrzej Cichocki, Shun-Ichi Amari

Adaptive Blind Signal and Image Processing

Learning Algorithms and Applications

• Gebundenes Buch

Produktbeschreibung

Im Mittelpunkt dieses modernen und spezialisierten Bandes stehen adaptive Strukturen und unüberwachte Lernalgorithmen, besonders im Hinblick auf effektive Computersimulationsprogramme. Anschauliche Illustrationen und viele Beispiele sowie eine interaktive CD-ROM ergänzen den Text.

---

Hinweis: Dieser Artikel kann nur an eine deutsche Lieferadresse ausgeliefert werden.

Produktdetails

• Verlag: Wiley & Sons
• 1. Auflage
• Seitenzahl: 586
• Erscheinungstermin: 14. Juni 2002
• Englisch
• Abmessung: 250mm x 175mm x 36mm
• Gewicht: 1142g
• ISBN-13: 9780471607915
• ISBN-10: 0471607916
• Artikelnr.: 09695820

Autorenporträt

Andrzej Cichocki received the M.Sc. (with honors), Ph.D. and Dr.Sc. (Habilitation) degrees, all in electrical engineering, from Warsaw University of Technology in Poland. Since 1972, he has been with the Institute of Theory of Electrical Engineering, Measurement and Information Systems, Faculty of Electrical Engineering at the Warsaw University of Technology, where he obtain a title of a full Professor in 1995. He spent several years at University Erlangen-Nuerenberg in Germany, at the Chair of Applied and Theoretical Electrical Engineering directed by Professor Rolf Unbehauen, as an Alexander-von-Humboldt Research Fellow and Guest Professor. In 1995-1997 he was a team leader of the laboratory for Artificial Brain Systems, at Frontier Research Program RIKEN (Japan), in the Brain Information Processing Group.

Inhaltsangabe

Preface xxix
1 Introduction to Blind Signal Processing: Problems and Applications 1
1.1 Problem Formulations - An Overview 2
1.1.1 Generalized Blind Signal Processing Problem 2
1.1.2 Instantaneous Blind Source Separation and Independent Component
Analysis 5
1.1.3 Independent Component Analysis for Noisy Data 11
1.1.4 Multichannel Blind Deconvolution and Separation 15
1.1.5 Blind Extraction of Signals 19
1.1.6 Generalized Multichannel Blind Deconvolution - State Space Models 20
1.1.7 Nonlinear State Space Models - Semi-Blind Signal Processing 22
1.1.8 Why State Space Demixing Models? 23
1.2 Potential Applications of Blind and Semi-Blind Signal Processing 24
1.2.1 Biomedical Signal Processing 25
1.2.2 Blind Separation of Electrocardiographic Signals of Fetus and Mother
26
1.2.3 Enhancement and Decomposition of EMG Signals 28
1.2.4 EEG and MEG Data Processing 28
1.2.5 Application of ICA/BSS for Noise and Interference Cancellation in
Multi-sensory Biomedical Signals 30
1.2.6 Cocktail Party Problem 35
1.2.7 Digital Communication Systems 36
1.2.8 Image Restoration and Understanding 38
2 Solving a System of Algebraic Equations and Related Problems 43
2.1 Formulation of the Problem for Systems of Linear Equations 44
2.2 Least-Squares Problems 45
2.2.1 Basic Features of the Least-Squares Solution 45
2.2.2 Weighted Least-Squares and Best Linear Unbiased Estimation 47
2.2.3 Basic Network Structure-Least-Squares Criteria 48
2.2.4 Iterative Parallel Algorithms for Large and Sparse Systems 49
2.2.5 Iterative Algorithms with Non-negativity Constraints 51
2.2.6 Robust Criteria and Iteratively Reweighted Least-Squares Algorithm 53
2.2.7 Tikhonov Regularization and SVD 57
2.3 Least Absolute Deviation (1-norm) Solution of Systems of Linear
Equations 61
2.3.1 Neural Network Architectures Using a Smooth Approximation and
Regularization 62
2.3.2 Neural Network Model for LAD Problem Exploiting Inhibition Principles
64
2.4 Total Least-Squares and Data Least-Squares Problems 68
2.4.1 Problems Formulation 68
2.4.2 Total Least-Squares Estimation 70
2.4.3 Adaptive Generalized Total Least-Squares 74
2.4.4 Extended TLS for Correlated Noise Statistics 76
2.4.5 An Illustrative Example - Fitting a Straight Line to a Set of Points
78
2.5 Sparse Signal Representation and Minimum 1-norm Solution 80
2.5.1 Approximate Solution of Minimum p-norm Problem Using Iterative LS
Approach 81
2.5.2 Uniqueness and Optimal Solution for Sparse Representation 84
2.5.3 FOCUSS Algorithms 84
3 Principal/Minor Component Analysis and Related Problems 87
3.1 Introduction 87
3.2 Basic Properties of PCA 88
3.2.1 Eigenvalue Decomposition 88
3.2.2 Estimation of Sample Covariance Matrices 90
3.2.3 Signal and Noise Subspaces - Automatic Choice of Dimensionality for
PCA 91
3.2.4 Basic Properties of PCA 94
3.3 Extraction of Principal Components 95
3.4 Basic Cost Functions and Adaptive Algorithms for PCA 99
3.4.1 The Rayleigh Quotient - Basic Properties 99
3.4.2 Basic Cost Functions for Computing Principal and Minor Components 100
3.4.3 Fast PCA Algorithm Based on the Power Method 102
3.4.4 Inverse Power Iteration Method 105
3.5 Robust PCA 105
3.6 Adaptive Learning Algorithms for MCA 108
3.7 Unified Parallel Algorithms for PCA/MCA and PSA/MSA 111
3.7.1 Cost Function for Parallel Processing 112
3.7.2 Gradient of J(W) 113
3.7.3 Stability Analysis 114
3.7.4 Unified Stable Algorithms 117
3.8 SVD in Relation to PCA and Matrix Subspaces 118
3.9 Multistage PCA for BSS 120
4 Blind Decorrelation and SOS for Robust Blind Identification 129
4.1 Spatial Decorrelation - Whitening Transforms 130
4.1.1 Batch Approach 130
4.1.2 Optimization Criteria for Adaptive Blind Spatial Decorrelation 132
4.1.3 Derivation of Equivariant Adaptive Algorithms for Blind Spatial
Decorrelation 133
4.1.4 Simple Local Learning Rule 136
4.1.5 Gram-Schmidt Orthogonalization 138
4.1.6 Blind Separation of Decorrelated Sources Versus Spatial Decorrelation
139
4.1.7 Bias Removal for Noisy Data 139
4.1.8 Robust Prewhitening - Batch Algorithm 140
4.2 SOS Blind Identification Based on EVD 141
4.2.1 Mixing Model 141
4.2.2 Basic Principles: SD and EVD 143
4.3 Improved Blind Identification Algorithms Based on EVD/SVD 148
4.3.1 Robust Orthogonalization of Mixing Matrices for Colored Sources 148
4.3.2 An Improved Algorithm Based on GEVD 153
4.3.3 An Improved Two-stage Symmetric EVD/SVD Algorithm 155
4.3.4 BSS and Identification Using a Bandpass Filters 156
4.4 Joint Diagonalization - Robust SOBI Algorithms 157
4.4.1 The Modified SOBI Algorithm for Nonstationary Sources: SONS Algorithm
160
4.4.2 Computer Simulation Experiments 161
4.4.3 Extensions of Joint Approximate Diagonalization Technique 162
4.4.4 Comparison of the JAD and Symmetric EVD 163
4.5 Cancellation of Correlation 164
4.5.1 Standard Estimation of Mixing Matrix and Noise Covariance Matrix 164
4.5.2 Blind Identification of Mixing Matrix Using the Concept of
Cancellation of Correlation 165
5 Statistical Signal Processing Approach to Blind Signal Extraction 177
5.1 Introduction and Problem Formulation 178
5.2 Learning Algorithms Using Kurtosis as a Cost Function 180
5.2.1 A Cascade Neural Network for Blind Extraction of Non-Gaussian Sources
with Learning Rule Based on Normalized Kurtosis 181
5.2.2 Algorithms Based on Optimization of Generalized Kurtosis 184
5.2.3 KuicNet Learning Algorithm 186
5.2.4 Fixed-point Algorithms 187
5.2.5 Sequential Extraction and Deflation Procedure 191
5.3 On-Line Algorithms for Blind Signal Extraction of Temporally Correlated
Sources 193
5.3.1 On-Line Algorithms for Blind Extraction Using a Linear Predictor 195
5.3.2 Neural Network for Multi-unit Blind Extraction 197
5.4 Batch Algorithms for Blind Extraction of Temporally Correlated Sources
199
5.4.1 Blind Extraction Using a First Order Linear Predictor 201
5.4.2 Blind Extraction of Sources Using Bank of Adaptive Bandpass Filters
202
5.4.3 Blind Extraction of Desired Sources Correlated with Reference Signals
205
5.5 A Statistical Approach to Sequential Extraction of Independent Sources
206
5.5.1 Log Likelihood and Cost Function 206
5.5.2 Learning Dynamics 208
5.5.3 Equilibrium of Dynamics 209
5.5.4 Stability of Learning Dynamics and Newton's Method 211
5.6 A Statistical Approach to Temporally Correlated Sources 212
5.7 On-line Sequential Extraction of Convolved and Mixed Sources 214
5.7.1 Formulation of the Problem 214
5.7.2 Extraction of Single i.i.d. Source Signal 215
5.7.3 Extraction of Multiple i.i.d. Sources 217
5.7.4 Extraction of Colored Sources from Convolutive Mixture 218
5.8 Computer Simulations: Illustrative Examples 219
5.8.1 Extraction of Colored Gaussian Signals 220
5.8.2 Extraction of Natural Speech Signals from Colored Gaussian Signals
222
5.8.3 Extraction of Colored and White Sources 222
5.8.4 Extraction of Natural Image Signal from Interferences 224
5.9 Concluding Remarks 224
6 Natural Gradient Approach to Independent Component Analysis 231
6.1 Basic Natural Gradient Algorithms 232
6.1.1 Kullback-Leibler Divergence - Relative Entropy as a Measure of
Stochastic Independence 232
6.1.2 Derivation of Natural Gradient Basic Learning Rules 235
6.2 Generalizations of the Basic Natural Gradient Algorithm 237
6.2.1 Nonholonomic Learning Rules 237
6.2.2 Natural Riemannian Gradient in Orthogonality Constraint 239
6.3 NG Algorithms for Blind Extraction 242
6.3.1 Stiefel and Grassmann-Stiefel Manifolds Approaches 242
6.4 Generalized Gaussian Distribution Model 244
6.4.1 Moments of the Generalized Gaussian Distribution 248
6.4.2 Kurtosis and Gaussian Exponent 250
6.4.3 The Flexible ICA Algorithm 250
6.4.4 Pearson System 254
6.5 Natural Gradient Algorithms for Non-stationary Sources 255
6.5.1 Model Assumptions 255
6.5.2 Second Order Statistics Cost Function 256
6.5.3 Derivation of Natural Gradient Learning Algorithms 256
7 Locally Adaptive Algorithms for ICA and their Implementations 273
7.1 Modified Jutten-H¿erault Algorithms for Blind Separation of Sources 274
7.1.1 Recurrent Neural Network 274
7.1.2 Statistical Independence 274
7.1.3 Self-normalization 277
7.1.4 Feed-forward Neural Network and Associated Learning Algorithms 278
7.1.5 Multilayer Neural Networks 281
7.2 Iterative Matrix Inversion Approach to the Derivation of a Family of
Robust ICA Algorithms 284
7.2.1 Derivation of Robust ICA Algorithm Using Generalized Natural Gradient
Approach 287
7.2.2 Practical Implementation of the Algorithms 288
7.2.3 Special Forms of the Flexible Robust Algorithm 290
7.2.4 Decorrelation Algorithm 290
7.2.5 Natural Gradient Algorithms 290
7.2.6 Generalized EASI Algorithm 290
7.2.7 Non-linear PCA Algorithm 291
7.2.8 Flexible ICA Algorithm for Unknown Number of Sources and their
Statistics 292
7.3 Blind Source Separation with Non-negativity Constraints 293
7.4 Computer Simulations 294
8 Robust Techniques for BSS and ICA with Noisy Data 305
8.1 Introduction 305
8.2 Bias Removal Techniques for Prewhitening and ICA Algorithms 306
8.2.1 Bias Removal for Whitening Algorithms 306
8.2.2 Bias Removal for Adaptive ICA Algorithms 307
8.3 Blind Separation of Signals Buried in Additive Convolutive Reference
Noise 310
8.3.1 Learning Algorithms for Noise Cancellation 311
8.4 Cumulant-Based Adaptive ICA Algorithms 314
8.4.1 Cumulant-Based Cost Functions 314
8.4.2 Family of Equivariant Algorithms Employing Higher Order Cumulants 315
8.4.3 Possible Extensions 317
8.4.4 Cumulants for Complex Valued Signals 318
8.4.5 Blind Separation with More Sensors than Sources 318
8.5 Robust Extraction of Arbitrary a Group of Source Signals 320
8.5.1 Blind Extraction of Sparse Sources with Largest Positive Kurtosis
Using Prewhitening and Semi-Orthogonality Constraint 320
8.5.2 Blind Extraction of an Arbitrary Group of Sources without
Prewhitening 323
8.6 Recurrent Neural Network Approach for Noise Cancellation 325
8.6.1 Basic Concept and Algorithm Derivation 325
8.6.2 Simultaneous Estimation of a Mixing Matrix and Noise Reduction 328
8.6.2.1 Regularization 329
8.6.3 Robust Prewhitening and Principal Component Analysis (PCA) 331
8.6.4 Computer Simulation Experiments for the Amari-Hopfield Network 331
9 Multichannel Blind Deconvolution: Natural Gradient Approach 335
9.1 SIMO Convolutive Models and Learning Algorithms for Estimation of a
Source Signal 336
9.1.1 Equalization Criteria for SIMO Systems 338
9.1.2 SIMO Blind Identification and Equalization via Robust ICA/BSS 340
9.1.3 Feed-forward Deconvolution Model and Natural Gradient Learning
Algorithm 342
9.1.4 Recurrent Neural Network Model and Hebbian Learning Algorithm 343
9.2 Multichannel Blind Deconvolution with Constraints Imposed on FIR
Filters 346
9.3 General Models for Multiple-Input Multiple-Output Blind Deconvolution
349
9.3.1 Fundamental Models and Assumptions 349
9.3.2 Separation-Deconvolution Criteria 351
9.4 Relationships Between BSS/ICA and MBD 354
9.4.1 Multichannel Blind Deconvolution in the Frequency Domain 354
9.4.2 Algebraic Equivalence of Various Approaches 355
9.4.3 Convolution as a Multiplicative Operator 357
9.4.4 Natural Gradient Learning Rules for Multichannel Blind Deconvolution
(MBD) 358
9.4.5 NG Algorithms for Double Infinite Filters 359
9.4.6 Implementation of Algorithms for a Minimum Phase Non-causal System
360
9.5 Natural Gradient Algorithms with Nonholonomic Constraints 362
9.5.1 Equivariant Learning Algorithm for Causal FIR Filters in the Lie
Group Sense 363
9.5.2 Natural Gradient Algorithm for a Fully Recurrent Network 367
9.6 MBD of Non-minimum Phase System Using Filter Decomposition Approach 368
9.6.1 Information Back-propagation 370
9.6.2 Batch Natural Gradient Learning Algorithm 371
9.7 Computer Simulation Experiments 373
9.7.1 The Natural Gradient Algorithm vs. the Ordinary Gradient Algorithm
373
9.7.2 Information Back-propagation Example 375
10 Estimating Functions and Superefficiency for ICA and Deconvolution 383
10.1 Estimating Functions for Standard ICA 384
10.1.1 What is an Estimating Function? 384
10.1.2 Semiparametric Statistical Model 385
10.1.3 Admissible Class of Estimating Functions 386
10.1.4 Stability of Estimating Functions 389
10.1.5 Standardized Estimating Function and Adaptive Newton Method 392
10.1.6 Analysis of Estimation Error and Superefficiency 393
10.1.7 Adaptive Choice of ' Function 395
10.2 Estimating Functions in Noisy Cases 396
10.3 Estimating Functions for Temporally Correlated Source Signals 397
10.3.1 Source Model 397
10.3.2 Likelihood and Score Functions 399
10.3.3 Estimating Functions 400
10.3.4 Simultaneous and Joint Diagonalization of Covariance Matrices and
Estimating Functions 401
10.3.5 Standardized Estimating Function and Newton Method 404
10.3.6 Asymptotic Errors 407
10.4 Semiparametric Models for Multichannel Blind Deconvolution 407
10.4.1 Notation and Problem Statement 408
10.4.2 Geometrical Structures on FIR Manifold 409
10.4.3 Lie Group 410
10.4.4 Natural Gradient Approach for Multichannel Blind Deconvolution 410
10.4.5 Efficient Score Matrix Function and its Representation 413
10.5 Estimating Functions for MBD 415
10.5.1 Superefficiency of Batch Estimator 418
11 Blind Filtering and Separation Using a State-Space Approach 423
11.1 Problem Formulation and Basic Models 424
11.1.1 Invertibility by State Space Model 426
11.1.2 Controller Canonical Form 428
11.2 Derivation of Basic Learning Algorithms 428
11.2.1 Gradient Descent Algorithms for Estimation of Output Matrices W=
[C;D] 429
11.2.2 Special Case - Multichannel Blind Deconvolution with Causal FIR
Filters 432
11.2.3 Derivation of the Natural Gradient Algorithm for the State Space
Model 432
11.3 Estimation of Matrices [A;B] by Information Back-propagation 434
11.4 State Estimator - The Kalman Filter 437
11.4.1 Kalman Filter 437
11.5 Two-stage Separation Algorithm 439
12 Nonlinear State Space Models - Semi-Blind Signal Processing 443
12.1 General Formulation of The Problem 443
12.1.1 Invertibility by State Space Model 447
12.1.2 Internal Representation 447
12.2 Supervised-Unsupervised Learning Approach 448
12.2.1 Nonlinear Autoregressive Moving Average Model 448
12.2.2 Hyper Radial Basis Function Neural Network Model (HRBFN) 449
12.2.3 Estimation of Parameters of HRBF Networks Using Gradient Approach
451
References 453
13 Appendix - Mathematical Preliminaries 535
13.1 Matrix Analysis 535
13.1.1 Matrix inverse update rules 535
13.1.2 Some properties of determinant 536
13.1.3 Some properties of the Moore-Penrose pseudo-inverse 536
13.1.4 Matrix Expectations 537
13.1.5 Differentiation of a scalar function with respect to a vector 538
13.1.6 Matrix differentiation 539
13.1.7 Trace 540
13.1.8 Matrix differentiation of trace of matrices 541
13.1.9 Important Inequalities 542
13.1.10Inequalities in Information Theory 543
13.2 Distance measures 544
13.2.1 Geometric distance measures 544
13.2.2 Distances between sets 544
13.2.3 Discrimination measures 545
14 Glossary of Symbols and Abbreviations 547
Index 552